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ABSTRACT 


This  study  Introduces  procedures  for  constructing  a  proficiency  scale 
for  a  large-scale  test  by  applying  Tatsuoka's  Rule  Space  Model.  The  SAT 
Mathematics  (SAT  M) ,  Section  2,  is  used  for  illustrating  the  process  and  the 
results.  A  task  analysis  is  summarized  in  a  mapping  sentence,  and  then  14 
processes  and  content  attributes  are  identified  for  explaining  the  underlying 
cognitive  aspects  of  the  examinees'  performance  on  the  SAT  M.  Analysis  results 
show  that  almost  98%  of  2334  examinees  are  successfully  classified  into  one  of 
468  cognitive  states.  The  cognitive  states  are  characterized  by  mastery  or 
non— mastery  of  the  14  attributes.  Attribute  Characteristic  Curves,  which  are 
conditional  probability  functions  defined  on  the  SAT  Scale,  are  introduced  and 
used  for  interpreting  an  examinees'  proficiency.  Prototypes  of  a  student's 
performance  report  and  a  group  performance  report  are  given  as  examples  of 
possible  ways  for  summarizing  the  analysis  results. 
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INTR0DUCTI(»1 


Recent  developments  in  cognitive  theories  have  shown  that  learning  is 
the  reorganization  and  integration  of  complex  tasks.  However,  learning  models 
considered  by  educational  measurement  are  primarily  linear,  and  hence 
measurement  models  that  have  been  developed  support  the  unidimensionality  view 
of  ability  levels.  The  purpose  and  goal  of  these  models  are  focused  on  making 
inferences  about  amount  of  ability  or  amovmt  of  knowledge  that  an  individual 
possesses,  which  can  be  located  on  the  continuum. 

A  new  view  of  achievement  that  emerges  from  cognitive  and  domain  studies 
emphasizes  the  importance  of  how  knowledge  is  organized,  what  processes  are 
used  to  solve  problems ,  the  degree  to  which  certain  procedures  and  processes 
are  automated,  and  the  ability  to  represent  knowledge  in  a  variety  of  ways. 

New  measurement  models  should  be  able  to  measure  such  abilities,  as  well  as 
traditional  ability  levels.  The  movement  for  searching  for  an  instructionally 
useful  way  of  assessing  students 's  performance  has  indicated  the  need  for  new 
measurement  theories  and  models.  The  movement  for  enhancing  the 
interpretability  of  test  scores  also  urges  one  to  develop  a  new  methodology  by 
which  test  users  with  different  interests  in  using  performance  results  would 
be  satisfied. 

Beaton  (1988)  introduced  a  method,  called  empirical  anchoring  and  applied 
it  to  the  NAEP  tests.  Rock  &  Johnson  (1989)  applied  this  method  to  the  SAT. 

The  method  starts  out  by  empirically  selecting  items  that  discriminate  between 
various  levels  on  the  total  score  distribution.  These  items  are  called 
"anchoring"  items.  Then  experts  review  the  anchor  items  that  describe  the 
skills  necessary  to  achieve  that  particular  score  level.  The  method  provides 
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empirical  probabilities  of  success  on  each  of  the  items  for  students  whose 
scores  were  near  the  anchoring  points  of  the  scale.  Although  this  method  has 
attracted  a  substantial  amount  of  attention  from  educators,  it  also  has 
invite  riticism  from  researchers  in  educational  measurement  and 
psychometrics  (Forsyth,  1991). 

Marco,  Crone,  Braswell,  Curley  and  Wright  (1990)  investigated  the 
relationship  between  SAT  content  variables  and  their  predictive  validity  and 
found  that  some  cognitive  tasks  are  important  for  predicting  students'  success 
in  their  future  performance . 

However,  test  item  development  has  been  atheoretical  in  terms  of 
cognitive  theory  (Gitomer,  1988).  It  is  important  to  understand  the  nature  of 
cognitive  processing  involved  in  SAT  Mathematics.  Gitomer  (1988)  pointed  out 
that  students'  errors  are  often  linked  to  an  inability  to  conceptualize  a 
problem,  to  a  failure  to  employ  efficient  problem-solving  heuristic,  and  to  a 
lack  of  willingness  to  pursue  difficult  problems  that  cannot  be  solved 
quickly . 

Schoenfeld  (1985)  argued  that  some  students  have  a  view  of  mathematics 
that  it  is  simply  equivalent  to  the  learning  of  algorithms.  However,  Gitomer 
(1988)  developed  a  diagnostic  test  that  was  designed  to  measure  knowledge, 
execution  referred  to  the  procedural  evaluation  of  a  problem  (such  as 
multiplying  two  polynomials) ,  application  involved  in  recognizing  a  procedure 
to  execute  for  a  given  problem,  decomposition  processes  that  require 
decomposing  a  problem  with  multiple  subgoals,  and  translation  (that  is,  the 
process  of  transferring  a  word  problem  into  a  representation  that  can  lead  to 
a  solution)  had  a  strong  relationship  with  mathematics  grades. 
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Enright  (1991)  emphasized  that  understanding  problem  solving  requires  a 
description  of  the  problem  as  well  as  a  description  of  problem  solving 
approaches  and  outcomes.  Gallagher  (1991)  investigated  sex  differences  on 
cognitive  tasks  for  SAT  Mathematics  and  found  that  female  students  tend  to  use 
algorithmic  strategies  as  test-taking  skills  while  male  students  tend  to  use  a 
systematic  trial-and-error  approach  regulated  by  some  unknown  reasoning.  These 
task  variables  are  useful  for  guiding  an  analysis  of  the  underlying  cognitive 
processes . 

A  task  analysis  of  the  SAT  Mathematics,  Form  8A,  was  performed  by  taking 
the  research  results  mentioned  above  into  account.  This  report  summarizes  the 
results  of  a  task  analysis  and  discusses  an  application  of  a  measurement  model 
called  Rule  Space  (Tatsuoka,  1983)  to  construct  a  descriptive  scale  for  SAT 
Mathematics.  The  approach  is  an  outcome  of  a  long-term  research  project 
supported  by  the  Office  of  Naval  Research,  and  the  model  actually  performs 
individual  diagnostic  analyses  of  examinees'  response  patterns.  The  results 
can  be  used  for  enhancing  learning,  improving  instruction,  and  remediation  of 
examinees'  weaknesses. 

The  model  projects  (or  converts)  examinees'  item  response  patterns  into 
their  performance  patterns  on  underlying  cognitive  tasks ,  which  are  identified 
by  a  task  analysis.  A  set  of  newly  converted  mastery  patterns  of  cognitive 
tasks  (called  attributes)  enables  one  to  estimate  conditional  probability 
functions  for  attributes  (PFAs)  on  the  SAT  Scale,  or  IRT  ability  scale  0. 

The  report  gives  some  tailored  prototypes  of  performance  reports 
suitable  to  various  interest  groups  of  test  users.  The  last  section  discusses 
the  generalizability  of  attributes  across  two  forms  of  the  SAT  M,  Section  2. 
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METHOD  AND  PROCEDURES 


1.  A  Task  Analysis  of  SAT  Mathematics 

A  description  of  the  process  that  led  to  the  specification  of  the 
attributes  employed  in  the  rule  space  analysis  is  described  in  this  Section. 

1.1.  A  mapping  sentence  The  cognitive  requirements  for  solving  the 
mathematics  items  of  Sections  2  and  5  of  SAT  (form  8B  administered  on  May  7, 
1988)  were  specified  using  data  from  two  protocols. 

In  order  to  siammarize  the  content  and  process  categories  identified  in 
the  protocol  analysis,  a  mapping  sentence  (Guttman,  1991;  Tziner,  1987)  was 
designed.  The  mapping  sentence  included  13  facets  with  a  varying  number  of 
elements  in  each.  Before  presenting  the  mapping  sentence,  a  word  of  caution  is 
in  order.  The  mapping  sentence  presented  in  Table  1.1  is  a  preliminary  one.  By 
no  means  do  we  contend  that  it  is  complete  or  exhaustive.  More  insight  into 

Insert  Table  1.1  about  here 

the  cognitive  requirements  underlying  the  SAT-M  items  needs  to  be  gained  by  a 
comprehensive  protocol  analysis  on  several  forms  of  the  SAT  before  a  complete 
cognitive  model  can  be  constructed. 

Every  item  in  the  test  can  be  expressed  as  a  combination  of  elements 
from  the  facets  of  the  mapping  sentence.  For  example:  Item  No.  1,  "If  2x  -  6  - 

10,  then  3x  -  6  - _  ,  (A)  0,  (B)  8,  (C)  11,  (D)  18,  (E)  24  "  can  be 

expressed  in  terms  of  the  above  mapping  sentence  as  the  following  combination 
of  facet  elements:  A3. 1.1,  Bl,  C2,  Dl,  E2,  FI,  Gl,  HI,  12.1,  J2,  K3.  L2,  Ml. 
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1.2.  Making  an  incidence  matrix  Twenty-seven  elements  from  the  mapping 
sentence  were  selected  and  expressed  as  attributes  to  be  used  in  the  initial 
rule  space  analysis.  Table  1.2.1  lists  these  attributes. 

Insert  Table  1.2.1  about  here 

An  incidence  matrix  Q  (60  items  by  27  attributes)  was  constructed  for 
SAT  Sections  2  and  5  using  the  above  mentioned  attributes.  Table  1.2.2 
presents  the  Q  matrix  al.  ag  with  the  percent  correct  responses  for  each  item 
and  values  of  the  IRT  item  difficulty  parameter  b. 

Insert  Table  1.2.2  about  here 

For  ease  of  referencing,  Table  1.2.3  lists  the  items  requiring  each  of 
the  27  attributes. 


Insert  Table  1.2.3  about  here 

1.3.  A  multiple  reeression  analysis  A  multiple  regression  analysis  was 
performed  to  predict  percent  correct  (of  60  items)  from  the  27  attribute 
vectors.  Table  1.3.1  presents  the  results  of  this  analysis. 

Insert  Table  1.3.1  about  here 

As  can  be  seen  in  Table  1.3.1,  83%  of  the  variance  in  item  difficulty 
(percent  correct)  was  explained  by  the  27  attributes.  Attributes  8,  19,  6,  3, 
2,  25,  27,  11,  21,  7,  4  had  the  highest  regression  weights.  The  negative 
signs  of  these  weights  indicate  that  the  presence  of  these  attributes 
contributes  to  the  items  being  more  difficult.  Attribute  15  had  a  relatively 
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high  positive  weight,  indicating  that  its  presence  is  associated  with  easier 
items. 

Based  on  the  regression  results,  the  initial  attribute  set  was  reduced  by 
collapsing  10  of  the  content  attributes  into  three  categories  and  omitting 
five  weak  attributes.  The  reduced  set  of  15  attributes  is  presented  in  Table 
1.3.2.  Table  1.3.3  lists  the  25  items  of  Section  2  by  the  reduced  set  of  14 
attributes.  (Attribute  16  is  relevant  to  Section  5  only.) 


Insert 

Table 

1.3.2 

about 

here 

Insert 

Table 

1.3.3 

about 

here 

1.4.  Analysis  of  SAT  M.  Section  2  The  incidence  matrix  Q  for  items  1-25  of 
Section  2  by  14  attributes  (see  Table  1.4.1)  was  subjected  to  multiple 
regression  analyses  for  predicting  item  difficulties  (percent  correct  and  IRT 
b-values).  The  results  of  the  two  regression  analyses  are  presented  in  Table 
1.4.2. 


Insert 

Table 

1.4.1 

about 

here 

Insert 

Table 

1.4.2 

about 

here 

As  can  be  seen  in  Table  1.4.2,  83%  and  91%  of  the  variance  in  item 
difficulty  (percent  correct  and  IRT  b-values,  respectively)  were  explained  by 
the  14  attributes.  In  both  analyses  the  strongest  attributes  were  Nos.  21,  19, 
17  and  25  (analytic  thinking;  comprehension  +  application;  understanding  of 
concepts;  and  multiple  steps  toward  the  solution). 
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The  Rule  Space  Model  has  recently  been  introduced  in  various  ETS 
technical  reports  (Tatsuoka  &  Tatsuoka,  1992;  Sheehan,  Tatsuoka  &  Lewis; 
Birenbaum,  Kelly  &  Tatsuoka,  1992).  So  a  brief  discussion  will  be  given  in  the 
next  section  and  Appendix  will  provide  a  more  detailed  sketch. 

2.  A  Brief  Discussion  of  the  Rule  Space  Model 

An  alternative  approach  to  cognitive  diagnosis  —  in  contrast  to  the 
traditional  bug  analyses  —  is  the  rule  space  model  which  is  a  probabilistic 
approach  whose  purpose  is  to  identify  the  examinees'  state  of  knowledge  or 
cognitive  states,  based  on  an  analysis  of  the  task's  cognitive  requirements. 

Having  specified  the  task's  cognitive  requirements  (also  called 
attributes),  an  incidence  matrix  Q  (K  x  n)  (the  number  of  attributes  x  the 
number  of  items)  is  constructed,  which  describes  item  characteristics  in  terms 
of  the  underlying  cognitive  processes  involved  in  each  item.  Cognitive 
patterns  represented  by  K  binary  elements  of  unobservable  attributes  that  can 
be  derived  from  the  incidence  matrix  Q  are  called  cognitive  states  (or 
attribute  patterns) .  Boolean  Descriptive  Functions  (BDFs)  are  used  to 
systematically  determine  these  cognitive  states  and  map  them  into  observable 
item  score  patterns  (called  ideal  item  score  patterns)  (Tatsuoka,  1991;  Varadi 
&  Tatsuoka,  1989) .  It  is  assumed  that  an  item  can  be  answered  correctly  if  and 
only  if  all  the  attributes  involved  in  the  item  have  been  mastered. 
Unobservable  performances  on  the  attributes  can  be  viewed  analogously  to  an 
unobservable  electric  current  running  through  various  switches  if  they  are 
closed.  A  closed  switch  corresponds  to  an  attribute  that  has  been  mastered. 

All  switches  in  a  circuit  must  be  closed  in  order  for  the  current  to  go 
through.  The  cognitive  states  are  represented  by  a  list  of  mastered/not 
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mastered  (or  "can/cannot")  attributes.  The  increase  of  the  number  of  states  is 
combinatorial,  but  Boolean  algebra  is  a  useful  tool  for  dealing  with  the 
problem  of  combinatorial  explosion.  Boolean  algebra,  which  has  been  widely 
used  for  explaining  various  properties  of  electricity  and  combinatorial 
circuits  have  been  utilized  within  the  rule  space  framework  for  explaining  the 
cognitive  requirements  underlying  test  performances. 

Once  the  cognitive  states  (ideal— item-score  patterns)  are  determined, 
the  actual  data  are  considered.  The  task  now  is  to  map  the  actual  item 
response  patterns  of  the  examinees  onto  the  cognitive  states,  i.e.,  to  find 
the  ideal-item-score  pattern  closest  to  the  student's  actual  response  pattern. 
Since  the  performance  on  test  items  usually  includes  slips  or  random  errors, 
the  observed  item-response  patterns  are  likely  to  deviate  to  some  extent  from 
the  ideal— item-s  ^re  patterns  represented  by  the  various  cognitive  states. 

Thus  one  is  faced  with  a  pattern  classification  problem  which  is  handled  by 
the  rule  space  model  (Tatsuoka  &  Tatsuoka,  1989) .  The  model  formulates  the 
classification  space  and  procedures.  Item  Response  Theory  (IRT)  is  utilized 
for  formulating  the  classification  space,  which  is  a  Cartesian  product  space 
of  IRT  ability  0  and  a  variable  f  which  measures  the  unusualness  of  item  score 
patterns  (Tatsuoka,  1984,  Tatsuoka  &  L:nn,  1983).  The  cognitive  states  as  well 
as  the  students'  item  response  patterns  are  mapped  as  points  in  the 
classification  space  by  computing  their  6  and  f  values.  Tatsuoka  (1990)  has 
shown  that  the  swarm  of  mapped  "fuzzy"  points  of  students'  item-response 
patterns  follows  approximately  a  multivariate  normal  distribution  with  the 
centroid  being  a  given  cognitive  state.  Bayes'  decision  rules  are  applied  for 
the  final  classification  and  for  computation  of  misclassif ication 
probabilities . 


10 


Once  this  classification  has  been  carried  out,  one  can  indicate  with  a 
specified  probability  level  which  attributes  a  given  examinee  is  likely  to 
have  mastered  or  failed  to  master.  If  classification  rates  are  as  high  as  80  % 
or  above,  then  the  attribute  mastery  patterns  can  be  used  for  statistical 
analyses.  For  example,  a  factor  analysis  can  be  applied  to  examine  the 
dimensionality  of  attributes,  or  a  discriminant  analysis  can  be  used  for 
investigating  subgroup  differences  if  the  demographic  information  is 
available.  Similar  to  the  estimation  of  Item  Response  Curves  from  the  item 
response  patterns,  it  is  possible  to  investigate  the  conditional  probability 
functions  of  the  attributes  defined  on  the  SAT  scale  or  IRT  6. 

3.  The  Classification  Results  of  SAT  M.  Section  2 

A  computer  program,  BUGLIB,  classified  2335  examinees  who  took  the  SAT 
M,  Form  8A,  into  one  of  600  cognitive  states.  Since  the  squared  Mahalanobis 
distance  in  this  case  follows  a  Chi-square  distribution  with  7  degrees  of 
freedom,  -  2.76  (p-.Ol)  is  set  as  the  first  criterion  for  whether  or  not  X 
can  be  classified  into  a  cognitive  state.  It  turned  out  that  98  %  of  the  2335 
examinees  qualified  according  to  the  first  criterion,  and  were  thus  classified 
into  one  of  600  cognitive  states.  The  examinees  who  were  not  classified  are 
mostly  very  high  scoring  students  and  their  0  values  are  larger  than  2.5. 

After  Bayes'  rule  was  applied  for  the  final  classification,  468  cognitive 
states  become  non-empty,  with  136  states  having  one  examinee  classified,  64 
states  having  2  classified,  32  states  having  3  classified,  26  states  having  4, 
14  states  having  5,  13  states  having  6,  13  having  7,  8  having  8,  and  5  having 
9.  The  states  to  which  at  least  11  examinees  were  classified  are  listed  in 
Table  3.1.  One  hundred  thirty  two  examinees  are 
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Insert  Table  3.1  about  here 

classified  into  State  472,  which  is  characterized  by  the  deficiency  of 
attributes  2,19,21,  and  25.  State  2,  which  is  characterized  by  the  lack  of 
skill  21,  has  180  examinees  classified. 

The  0-values  and  f-values  for  the  cognitive  states  which  are  listed  in 
Table  3.1  are  given  in  Table  3.2. 

Insert  Table  3.2  about  here 

Table  3.2  indicates  some  interesting  trends  for  the  lack  of  skills 
across  various  levels  of  6.  For  example,  the  low-ability  examinees  missed 
Attributes  1,  3  and  21  (Arithmetic,  advanced  algebra  and  analytical  thinking 
skill)  while  high-ability  examinees  missed  Attribute  21  and  could  do  most 
content  areas  except  for  advanced  algebra.  Probability  Functions  for  the 
attributes  (PFAs)  will  provide  us  trends  of  the  14  attributes  across  0. 

However,  before  discussing  PFAs,  simple  descriptive  statistics  of  the  14 
attributes  are  summarized.  Table  3.3  shows  the  simomary  statistics  of 

Insert  Table  3.3  about  here 


the  14  attributes  and  0,  f  and  five  generalized  fs.  Attributes  21,  19,  and  3 
are  difficult  attributes  while  Attributes  18,  6,  15,  and  23  are  easy  ones. 

The  means  of  0,  f  and  five  generalized  fs  are  closer  to  zero  and  the  standard 
deviations  are  almost  1  as  their  theoretical  means  and  standard  deviations 
indicate.  The  correlations  of  0  with  the  14  attributes  range  from  .05 
(Attribute  23)  to  .30  (Attribute  3).  The  correlations  of  f  with  the  14 
attributes  are  between  .14  (Attribute  19)  and  -.34  (Attribute  24),  except  for 
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that  of  Attribute  21  which  is  .58.  The  value  .58  indicates  that  the  behavior 
of  Attribute  21  is  unusual,  and  examinees  with  unusual  response  patterns  tend 
to  have  the  mastery  score  of  one  for  this  attributes.  The  dimensionality  of 
the  14  attributes  is  tested  by  computing  the  eigenvalues  of  the  correlation 
matrix  of  14  attributes.  The  results  of  Principal  Component  analysis  indicated 
that  the  14  attributes  are  not  unidimensional.  Of  course  we  could  have 
examined  the  dimensionality  with  better  statistics  such  as  Stout's  method 
(Stout,  1987),  but  we  will  leave  it  for  a  future  work. 

4.  Probabilities  for  Attributes 

When  examinees'  item  response  patterns  are  classified  into  particular 
states,  their  corresponding  attribute  mastery  patterns  are  then  known.  We  use 
the  attribute  mastery  patterns  to  estimate  probability  functions  for  the 
attributes  (PFAs).  PFAs  are  the  conditional  probability  functions  defined  on 
9,  and  they  describe  the  basic  characteristics  of  the  behavior  for  the 
attribute  variables.  By  looking  at  the  graphs  of  PFAs,  one  can  see  the 
relationships  between  the  performances  on  the  attributes  and  the  IRT  0-scale 
or  SAT  scale.  Each  attribute  should  have  its  unique  curve,  different  from 
those  of  the  others.  By  comparing  two  cuirves,  one  can  see  which  attribute  is 
harder.  They  may  Intersect  at  some  point,  with  abscissa  6q.  In  that  case  there 
is  an  interaction  between  item  difficulty  and  ability  level  exists.  Unlike 
Item  Response  Theory,  we  do  not  restrict  the  possible  forms  of  the  conditional 
probability  functions  by  assuming  that  they  belong  to  a  prespecified  family  of 
parametric  functions  such  as  logistic  or  noirmal  ogive.  Since  our  intention  is 
to  "let  the  data  speak  for  themselves,"  a  nonparametric  estimation  approach  is 
adopted  in  this  report. 
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attributes  Non-parametric  estimation  of  the  unknown  density  function  f  from 
a  plot  of  frequencies,  the  histogram,  has  been  well  investigated  by  many 
statisticians  (Hardle,  1991;  Scott,  1985),  Several  psychometricians  have 
applied  these  techniques  to  estimating  Item  Response  Curves,  which  are  not 
density  functions  (Ramsay,  1991;  Mokken  &  Lewis,  1982,  Lewis,  1990). 

Instead  of  plotting  a  histogram  of  observed  frequencies,  an  PFAs  for  a 
particular  attributes  constructed  by  first  classifying  examinees  into  bins  bj 
based  on  their  estimated  6  values  and  then  computing  the  proportion  of 
examinees  in  each  bin  who  have  been  classified  as  having  mastered  the 
attribute.  These  proportions  are  then  plotted  against  0  and  smoothed. 
Alternatively,  examinees  may  be  classified  into  bins  based  on  their  SAT  Scale 
scores.  The  PFA  would  then  be  plotted  as  a  function  of  the  SAT  Scale  score. 

4.2  Results  Using  SPLUS  on  a  SUN  SPARC  station,  a  computer  program  for 
estimating  Attribute  was  written.  In  the  program,  examinees  were  classified 
into  one  of  12  bins  based  on  their  SAT  Scale  score.  Figure  4.1  contains  the 
resulting  PFAs  for  each  of  the  14  attributes. 

Insert  Figure  4.1  about  here 

Tht  cuirves  in  Figure  4.1  are  not  well  smoothed  yet,  but  they  should 
suffice  for  the  purpose  of  introducing  the  concept  of  PFAs  for  an  attribute 
variable  to  the  reader  of  this  report.  Improved  methods  for  estimation  of  PFAs 
and  estimation  of  confidence  intervals  will  be  given  in  a  future  report. 


14 


4.3  Interpretation  Once  examinees'  SAT  Scale  scores  are  known,  their 
probabilities  of  mastering  each  of  the  attributes  can  be  read  off  the  curves 
given  in  Figure  4.1.  As  an  example.  Table  4.3.1  provides  attribute  mastery 
probabilities  for  the  first  eight  examinees  in  the  data  set. 

Insert  Table  4.3.1  about  here 

Examinee  5  has  a  very  high  SAT  Scale  score,  and  he/she  is  doing  very 
well  on  most  attributes  except  for  3,17,19,  and  21.  His  SAT  Scale  score  is 
almost  as  high  as  Examinee  7 ,  but  his  attribute  sores  are  much  lower  for  17 , 

19,  and  21.  By  looking  at  the  profile  of  each  student,  one  can  get  useful 
information  for  remediation  planning.  Alternatively,  by  looking  at  the  unit  of 
classrooms  or  schools,  one  can  make  useful  curriculum  design,  or  evaluation  of 
the  past  instruction  or  planning. 

4.4  Percentile  scores  Mokken  &  Lewis  (1982)  developed  a  non-parametric , 
Bayesian  IRT  model  which  is  based  on  the  Mokken-scale ,  and  Lewis  (1990) 
developed  an  algorithm  for  estimating  the  x%  threshold  for  a  monotone 
regression  function.  His  program  MonoReg2  (1990)  computes  the  posterior  mean 
estimate  of  a  percent  point  of  interest.  For  example.  Attribute  19  has  546  for 
the  50%  point,  277  for  25%  point  and  760  for  the  75%  point. 

Inseri.  Figure  4.4.1  about  here 

Figure  4.4.1  shows  the  empirical  curve  for  Attribute  19  and  posterior 


median  estimates  of  selected  values  of  the  corresponding  theoretical  function 
(connected  by  straight  lines).  With  this  method,  a  desired  percent  point  and 


its  corresponding  SAT  scale  score  can  be  obtained.  A  summary  table  could  then 
be  prepared,  describing  the  location  of  the  attribute  on  the  SAT  scale. 

4.5.  Enhancing  score  reports  Enhancing  a  score  report  can  be  done  by 
utilizing  the  probability  of  successful  performance  on  each  attribute, 
together  with  the  information  obtainable  from  item-level  analyses  such  as 
computing  IRT  conditional  probabilities  on  6.  The  incidence  matrix  Q  can  be 
used  to  retrieve  a  meaningful  subset  of  items  that  involves,  say,  "test  taking 
skills"  or  "higher  level  thinking  skills".  Therefore,  the  results  from  the 
rule-space  model  can  be  used  for  preparing  a  variety  of  reports  that  are 
tailored  to  different  groups  of  test  users.  The  purposes  for  using  the  test 
reports  may  vary  among  different  groups  of  test  users. 

The  optimal  use  of  test  results  should  be  recommended.  If  the  audience 
is  higher  educational  institutes,  test  results  are  used  for  selection  or 
placement  of  applicants.  Individual  examinees  in  high  schools  may  use  test 
results  for  guiding  themselves  for  further  study  or  remediation,  and  teachers 
for  evaluating  their  instructions,  for  designing  of  curricula  and  future 
instruction  planing.  The  test  results  can  also  be  used  for  preparing  reports 
for  group  performance.  Summary  statistics  of  attribute-level  performance  as 
well  as  item-level  performance  can  be  useful  for  schools ,  for  various 
districts  and  state  offices  of  education.  The  following  figure  gives  an 
example  of  what  we  can  offer  to  the  test  users. 

Insert  Figure  4.5.1  about  here 

The  data  banks  available  for  enhancing  scoring  reports  consist  of  four 
parts:  1)  The  score  matrix,  each  row  of  which  contains  a  student  ID,  an  item 
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response  pattern,  a  6— value,  a  value  (an  index  for  measuring  atypicality  of 
a  response  pattern)  and  an  attribute-mastery  pattern;  2)  the  incidence  matrix; 
3)  the  probability  matrix  for  indicating  each  item's  success  rate  at  various 
levels  of  8  and  SAT  scale;  and  4)  the  probability  matrix  for  indicating  each 
attribute's  mastery  rate  at  various  levels  of  6  and  SAT  scale.  The  information 
mentioned  above,  together  with  demographic  information  can  provide  test  users 
with  a  variety  of  reports  tailored  to  different  groups  based  on  their  needs 
and  interests.  The  following  figures  show  prototypes  of  reports  that  can  be 
assembled  from  the  database  (see  Appendix). 

Figures  4.5.2,  and  4.5.3 

Figures  4.5.2  and  4.5.3  are  prepared  for  examinees  who  are  interested  in 
understanding  their  weaknesses  and  strengths,  while  Figure  4.5.4  is  for  a 
class  room  teacher  who  is  interested  summary  statistic  and  class  evaluation. 
Rearranging  the  probability  matrix  by  the  order  of  total  scores  and  item 
difficulties  enables  teachers  and  administrators  to  identify  possible  problem 
areas  (Birenbaum,  1992). 

5.  Are  the  14  Attributes  Invariant  Across  Different  Forms  of  SAT  Mathematics  ? 

A  replication  study  was  carried  out  by  applying  the  14  attributes  to  a 
different  SAT  form  (OA  March,  1990).  Table  5.1  presents  the  incidence  matrix 

Insert  Table  5 . 1  about  here 


for  the  25  items  of  Section  2  of  that  form  by  the  14  attributes,  along  with 
the  item  difficulties  (percent  correct) . 


Insert  Table  5.2  about  here 

Table  5.2  presents  the  regression  results  for  predicting  item  difficulties  of 
the  25  items  of  Form  OA  (section  1}  from  the  14  attributes. 

As  can  be  seen  in  the  table,  91%  of  the  variance  in  item  difficulty 
(percent  correct)  was  explained  by  the  14  attributes.  The  strongest  attributes 
were  Nos.  3,  21  20  and  25  (advanced  algebra;  analytic  thinking;  reasoning;  and 
multiple  steps  toward  the  solution,  respectively). 

Upon  reviewing  the  items  of  Form  OA  an  additional  attribute  was 
introduced  to  the  original  set,  namely.  Attribute  26  "changing  the  unit  of 
measurement".  That  attribute  appeared  in  items  10  and  18  of  Form  OA,  Section 
2.  The  incidence  matrix  Q  for  the  15  attributes  appears  in  Table  5.3, 

Insert  Table  5.3  about  here 

For  ease  of  referencing,  Table  5.4  lists  the  attributes  involved  in  each 
of  the  25  items  (Form  OA,  Section  2)  and  Table  5.5  lists  the  items  that 
involve  each  of  the  15  attributes. 

Insert  Table  5.4  and  5.5  about  here 

A  regression  analysis  of  the  incidence  matrix  with  the  additional  attribute 
(No.  26)  is  presented  in  Table  5.6. 

Insert  Table  5.6  about  here 

As  can  be  seen  in  the  table,  94%  of  the  variance  in  item  difficulties  is 
explained  by  the  15  attributes.  The  strongest  attributes  in  this  analysis  are: 


18 


26,  3,  20  ,6,  and  25  (changing  the  unit  of  measurement;  advanced  algebra; 
reasoning;  elementary  geometry;  and  multiple  steps  toward  the  solution). 

This  routine  multiple  regression  analysis  suggests  Chat  the  attributes 
valid  for  one  form  may  be  valid  for  another  form.  However,  it  does  not  give 
any  direct  Information  for  assurance  chat  an  estimated  PFA  for  Attribute  A^ 
involved  in  one  form  will  be  very  close  to  the  estimated  PFA  from  a  different 
form.  If  the  construction  of  parallel  test  forms  were  to  be  based  on  the 
matching  of  attributes  across  different  forms,  Chen  our  concern  for  invariance 
of  PFAs  across  the  forms  may  not  be  so  important.  However,  the  current 
practice  of  test  construction  procedures  do  not  consider  the  underlying 
cognitive  attributes  of  test  performance.  The  procedures  emphasize  matching  of 
content  domains  although  SAT  Mathematics  tests  is  designed  for  measuring 
reasoning  rather  than  for  measuring  the  competency  in  content  domains . 

Discussion 

The  influence  of  SAT  Verbal  and  Mathematics  tests  on  American  education 
is  so  noticeable  that  maximizing  the  amount  of  information  obtainable  from  the 
test  scores,  and  searching  for  ways  to  utilize  such  information  optimally  are 
very  important.  This  study  introduced  a  new  way  to  construct  a  proficiency 
scale  by  applying  the  rule  space  model. 

The  rule  space  model  is  a  symbolic  parametric  model  in  which  the 
performances  on  unobservable  cognitive  tasks  are  inferred  from  observable  item 
scores.  The  inferred  attribute-mastery  patterns  are  used  for  estimating 
Attribute  Characteristic  Curves  defined  on  the  0  or  SAT  scale.  The  proficiency 
scale  in  this  paper  is  derived  from  these  PFAs . 


Statistical  matters  such  as  construction  of  confidence  intervals  for 
PFAs  and  further  improvement  of  non-par ame trie  estimation  methods  are  not 
discussed  in  this  paper.  The  technical  aspect  of  obtaining  percentile  scores 
from  PFA  should  also  be  sought  in  a  future  paper.  A  multidimensional  rule 
space  has  been  introduced  for  the  first  time  in  this  paper,  but  technical 
details  of  the  multidimensional  space  will  be  discussed  elsewhere  in  the  near 
future . 

A  list  of  the  14  attributes  should  be  examined  more  carefully  before  the 
proficiency  scale  for  SAT  M  is  to  be  used  in  practice.  The  regression  analysis 
and  the  rule-space  classification  don't  necessarily  provide  the  best  unique 
set  of  attributes.  Instead,  they  can  indicate  whether  or  not  these  attributes 
provide  a  useful  representation  of  the  underlying  cognitive  processes  of  the 
test.  There  may  exist  other  sets  of  attributes  that  are  as  good  as  the 
original  14  attributes.  Further  investigation  on  the  determination  of  the 
optimal  set  of  attributes  is  needed. 
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Table  1.1 


A  Mapping  Sentence  For  S  AT-M 


In  order  to  solve  item  x,  which  represents  s  task  with  the  following  characteristics: 

A 

content 

'  1.  Arithmetic  rl)  basic  operations  with  whole  numbers 

2)  signed  numbers  operations 

3)  finctions.  decimals 
14)  square  root,  exponents 

2.  Mathematics  1)  properties  of  numbers,  combinatorial,  inequality 
unit  of  measuremem 


] 


3.  Algebra 


1)  basic 
1 2)  advanced 


fd) 
1(2) 


4.  Geometry  rl)  elementary 


‘  S.  Statistics 


analytic 

Probability 


fS!- 

/(l)  lin 

1(2)  iria 

X3)  dr 


linear  equations 
simultaneous  linear  I  ^ 
quadratic  eql  V 

s-.,  functions  n 
(1)  lines,  rectangles 
triangles 
circles 


0 


B 

context 

regular  math  7 

2.  quantitative  comparisons 


seuing  ^ 
concrete/ 
abstract) 


F 

Q.  Structure 

{Logic  (if.,.then) 


G 

answer  ^pe 

1.  exact  number 

2.  ipptoximation^ 

3.  variable 


w  question  type 
Jl.  routine  7 
i2.  non  routineV 


H 


language  of  presentation 
1.  verbal  (word  problems)  Jl)  realistic  context  ( 


response  format 
Cl.  multiple  choice 

2.  constructed  (grid) 

3.  stem  includes  options  I 


numerical 

spatial 


suscqrdbillty  to 
(1.  low 


p)  imaginary  contex^ 


test  wiseness 


high  V  1}  options  can  be  used  to  get  die  answer  1 
t  2)  can  be  solved  intuitively  /  by  examplel  ^ 


visual  solution  possible 


Jj 


and  udiich  the  solution  process  involves: 


no.  of  steps 
|1.  one  1 
[2.  two  \ 

13.  thre^ 


requiring  to  read 
charts 
figures 
math  notati( 


4i 


L 

calculator 
.  rwt  needed 
.  can  be  helpful 
.  needed 


the  examinee  has  to  demonstrate  the  following: 


M 


5. 


6. 


Processes 

Application  of  simple  rulesfalgorithms  (perform  computations) 

Comprehension  +  application  of  rules/theorems/definitions/^rincipals/laws 
Translation  from  one  mode  to  another 
Creation  of  an  equation  with  Jl)  one  unknown 

{\,2)  more  than  one  unknownf 
1)  decomposition  of  a  simple  problem  \  and  restructuring  | 
2)  decomposition  of  a  complex  problem/ 

Reading  comprehension  J 1)  general  1 

|2)  spedfic  terminology) 


Table  1.2.1 


S  AT-M  27  Attributes 


Attribute  Attribute’s  Description 

No. 


A.  Content  related  attributes 

1 .  Arithmetics  (■»-  -  X : ;  signed  #s;  #  line;  ( );  factmng,  prq)erties  of  #s; 

combinatorial). 

4.  Arithmetics  •  fractions  (-t-  ratio;  decimals;  probability;  %) 

5.  Arithmetics  -  exponents  (+ sq.  root). 

22.  Arithmetic  •  inequality. 

2 .  Algebra  -  linear  equations  (->■  simultaneous  linear). 

3.  Algebra  -  quadratic  equations. 

27.  Algebra  -  Functions  (-f  relationships  between  number  and  symbols). 

6.  Geometty  •  li^  rectangles. 

7.  Geometry  -  triangles. 

8.  Geometry  -  Qrcles. 

26.  Analytic  gemnetty/reading  charts. 

9.  Measurement  reli^  concq)ts. 

10.  Notiroutine  problems  (nonconventional). 

1 1.  Language  of  presentatiaa:  Veibal  (Word  problem). 

12.  Language  of  presentation:  Numerical  (mkh  notations) 

13.  Language  of  {xesentation:  V  *f  Spatial  (figure  given). 

14.  Language  of  presentation:  V  +  Spatial  (figure  to  be  drawn). 

15.  Logic  (if...then). 

16.  Quantitative  ctnnparisons. 

B.  Process  Related  Attributes 


17.  Understanding  of  the  meaning  of  concepts. 

1 8.  Application  of  simple  rules/algorithms  (SOLVE:  perform  con^)utations). 

19.  Con^hension  +  tq>plication  of  rules/AeOTems  (chooses  and  applies 
coiiectly). 

20.  Reastxiing  (creates  an  equation). 

2 1 .  Analytic  inking,  copiitive  restructuring  ^igher  mental  processes). 

23.  Rea^g  comprehension  (+  follow  insrnicticms;  matii/geometry  terminoto^). 

24.  Test-wiseness  (solves  intuitively;  by  example;  goes  backwards  fiom  the  given 
answers). 

25.  Number  of  steps  in  the  solution  >  1 
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laoie  I.Z.Z 


biddefioe  Matrix  Q  for  27  AtttfiHiies  and  60  SAT-M  Items 
Attributes 

hem  .  111111111122222222  %  b 

No.  12345678901234S678901234S67  Gontct  Value 


01 

0 

1 

0 

0 

0 

0  0  0 

0  0 

E 

0  10  0  1 

0  0  0 

0  0 

0 

1  0 

0 

81  -1.513 

02 

0 

0 

0 

1 

1 

0  0  0 

0  0 

0 

\ 

0  0  0  0  1 

0  0  0 

0  0 

0 

0  0 

0 

89  -2.037 

03 

1 

0 

0 

0 

0 

0  0  0 

0  0 

0 

0  0  0  0  1 

0  0  0 

0  1 

0 

1  1 

0 

88  -1.554 

04 

1 

0 

0 

0 

0 

0  0  0 

0  0 

0 

0  0  0  0  1 

0  0  0 

0  0 

0 

1  1 

0 

76  -1.214 

05 

0 

0 

0 

1 

0 

0  0  0 

0  0 

0 

1 

0  10  0  1 

0  0  0 

0  0 

1 

0  0 

0 

80  -1.021 

06 

0 

0 

0  0 

1 

0  0  0 

0  0 

0 

1 

0  10  0  1 

0  0  0 

0  0 

0 

0  0 

0 

85  -1.371 

07 

0 

0 

0 

0 

0 

0  10 

0  0 

0 

0  0  0  0  0 

0  10 

0  1 

1 

0  0 

0 

72  -2.916 

08 

1 

0 

0 

0 

0 

0  0  0 

0  1 

1 

0  10  0  0 

0  10 

0  1 

1 

0  0 

0 

59  -0.538 

09 

1 

0 

0 

E 

0 

0  0  0 

0  0 

0 

1 

0  0 

0 

0  0 

0 

76  -1.056 

EH 

0 

0 

0 

1 

0 

0  10 

0  0 

0 

10  0 

0  0 

0 

1  0 

0 

53  .157 

11 

1 

1 

0 

1 

0 

0  0  0 

0  1 

1 

0  10  0  1 

0  10 

0  0 

0 

0  0 

0 

60  -.433 

12 

■  ( 

0 

0 

0 

0 

0  0  0 

1  1 

1 

« 

0  0  0  0  1 

0  0  0 

0  0 

0 

0  0 

0 

64  -.218 

13 

K 

0 

1 

1 

1 

0  0  0 

0  0 

0 

i 

0  10  0  1 

0  0  0 

0  0 

0 

0  0 

0 

58  -.449 

14 

0 

0 

E 

0 

10  0 

0  0 

0 

1 

110  10 

0  0  0 

0  1 

0 

0  0 

0 

72  -1.190 

15 

Mi 

0 

0 

1 

0 

0  0  0 

0  0 

0 

« 

0  10  0  1 

0  0  0 

0  0 

0 

1  0 

0 

48  .777 

16 

1 

0 

0 

E 

0 

0  0  0 

0  1 

1 

1 

IiMiMiMiM*] 

0  10 

0  0 

0 

1  0 

0 

55  .328 

17 

0 

0 

0 

0 

0 

10  0 

0  0 

0 

0  0 

1  0  0  0  0 

0  0  1 

0  1 

0 

1  0 

0 

38  .619 

18 

0 

0 

0 

0 

0 

0  0  1 

0  0 

0 

0  1 

0  10  0  0 

110 

0  0 

0 

1  0 

0 

35  .881 

19 

0 

1 

0 

1 

0 

0  0  0 

0  1 

1 

0  0 

0  10 

0  0 

0 

1  0 

0 

33  .842 

20 

1 

0 

0 

0 

0 

0  0  0 

0  1 

1 

0  0 

0  0  0  0  0 

0  10 

0  1 

1 

1  0 

0 

32  .805 

21 

0 

0 

0 

0 

1 

0  0  0 

0  0 

0 

1  0 

0  0  0  1  1 

0  0  0 

0  1 

0 

1  0 

1 

27  .840 

22 

0 

0 

0 

0 

0 

0  0  1 

0  0 

1 

0  0 

0  10  0  1 

0  10 

1  1 

0 

1  0 

0 

23  .899 

23 

0 

1 

0 

0 

0  0  0 

0  1 

1 

0  0 

0  0  0  0  1 

oil 

0  1 

0 

1  0 

0 

19  1.534 

24 

0 

0 

0 

0 

0 

110 

0  0 

0 

0  1 

0  10  0  1 

111 

0  0 

0 

1  0 

0 

14  1.788 

25 

0 

1 

0 

1 

0 

0  0  0 

1  1 

0 

0  1 

0  0  0  0  1 

oil 

0  0 

0 

1  0 

0 

17  1.351 

26 

1 

0 

0 

0 

0 

0  0  0 

0  0 

0 

1  0 

0  0  0 

0  0 

0 

0  0 

0 

92  -2.977 

27 

1 

0 

0 

□ 

0 

0  0  0 

0  0 

1 

FhI 

0  10  0  1 

0  10 

0  0 

0 

0  0 

0 

92  -1.888 

28 

0 

0 

0 

0 

0 

0  0  0 

1  1 

1 

0  0 

0  10  0  0 

0  10 

0  0 

1 

0  0 

0 

80  -2.466 

29 

0 

0 

0 

0 

1 

0  0  0 

0  0 

0 

1  0 

0  0  0  0  1 

0  0  0 

0  0 

1 

0  0 

0 

79  -1.265 

30 

0 

0 

0 

0 

0 

10  0 

0  1 

0 

0  1 

0  0  0  0  0 

0  0  0 

0  1 

0 

0  0 

0 

68  -.662 

31 

0 

0 

0 

0 

0 

0  0  0 

0  1 

0 

1  0 

0  10  10 

0  10 

0  1 

0 

0  0 

1 

75  -1.182 

32 

0 

0 

0 

0 

0 

10  0 

0  1 

0 

0  1 

0  0  0  1  0 

0  0  0 

0  1 

0 

0  0 

0 

64  -.765 

33 

1 

0 

0 

1 

0 

0  0  0 

0  0 

0 

1  0 

0  0  10  0 

0  0  0 

1  0 

0 

0  0 

0 

74  -.217 

34 

0 

□ 

0 

□ 

0 

0  10 

0  0 

1 

0  0 

0  0  110 

0  0  0 

0  0 

0 

0  0 

0 

76  -.920 

35 

0 

0 

0 

1 

0 

0  0  0 

1  0 

0 

Esa 

0  0  110 

0  0  0 

0  0 

0 

0  0 

0 

67  -.511 

36 

0 

0 

0 

1 

0 

0  0  0 

0  0 

0 

1  0 

0  0  10  1 

0  0  0 

0  0 

0 

0  0 

0 

72  -.372 

37 

0 

0 

0 

0 

0 

10  0 

0  0 

0 

0  1 

0  0  10  1 

0  0  0 

0  0 

0 

0  0 

0 

65  -.215 

36 

1 

0 

0 

0 

0 

0  0  0 

0  0 

0 

0  1 

0  0  10  0 

0  0  0 

0  0 

0 

0  1 

0 

83  -1.437 

39 

1 

0 

0 

0 

0 

0  0  0 

0  0 

0 

0  1 

EESEEj 

0  0  0 

0  0 

0 

1  1 

0 

73  -1.246 

40 

1 

0 

0 

0 

0 

0  0  0 

0  0 

0 

1  0 

0  0  10  1 

0  0  0 

0  0 

0 

0  0 

0 

79  -1.081 

41 

1 

0 

0 

0 

0 

0  0  0 

0  1 

1 

0  0 

0  0  111 

0  0  0 

0  1 

0 

1  0 

0 

49  .175 

42 

1 

0 

0 

0 

0 

0  0  0 

0  1 

1 

0  0 

0  0  111 

0  0  0 

0  1 

0 

1  0 

0 

45  .254 

43 

□ 

1 

0 

0 

0 

0  0  0 

0  c 

0 

1  0 

0  0  10  1 

0  0  0 

0  0 

0 

0  0 

0 

67  -.741 

44 

1 

0 

0 

El 

0 

0  0  0 

0  1 

0 

1  0 

0  0  10  0 

0  0  0 

0  1 

0 

0  0 

0 

61  -.509 

45 

0 

0 

0 

1 

0 

0  0  0 

0  0 

0 

1  0 

0  0  110 

0  0  0 

1  0 

0 

0  0 

0 

56  -.104 

46 

0 

0 

0 

0 

0  10 

0 

0  0  10  1 

0  0  0 

0  0 

0 

0  0 

0 

61  -.094 

47 

0 

0 

0 

1 

0 

0  0  0 

zl 

1 

0  0  110 

0  0  0 

0  1 

0 

0  0 

0 

49  .543 

46 

0 

0 

0 

1 

0 

0  0  0 

0 

1  0 

0  0  110 

0  0  0 

0  0 

0 

0  0 

0 

35  .985 

49 

0 

0 

0 

0 

0 

10  0 

0  0 

0 

PWl 

10  110 

0  0  1 

0  1 

0 

1  0 

0 

26  2.042 

0 

1 

0 

0 

0  0  0 

0  1 

0 

1  0 

0  0  10  1 

0  0  1 

0  0 

0 

1  0 

0 

28  1.303 

51 

0 

1 

1 

0 

0 

0  0  0 

0  0 

0 

1  0 

0  0  10  0 

0  0 

1 

0  0 

0 

30  1.378 

52 

0 

0 

0 

0 

0 

110 

rwii 

0 

nwil 

10  10  0 

ESZl 

0  0 

0 

1  0 

0 

09  1.939 

53 

0 

0 

0 

0 

0 

10  0 

Bl 

1 

0  0 

0  10  0  1 

0  10 

0  0 

0 

0  0 

0 

57  .577 

54 

1 

1 

0 

0 

0 

0  0  0 

0  1 

1 

0  0 

0  10  0  0 

0  0  0 

1  1 

1 

0  0 

0 

54  -.147 

55 

0 

0 

0 

0 

0 

10  0 

0  1 

0 

0  0 

10  0  10 

0  0  1 

0  0 

0 

1  0 

0 

41  1.150 

56 

1 

0 

0 

0 

0 

0  0  0 

0  0 

1 

0  1  0  0  t) 

oil 

0  1 

1 

1  1 

0 

22  1.286 

57 

0 

1 

0 

0 

0 

0  10 

0  0 

0 

0  1 

0  0  0  0  1 

10  0 

0  0 

0 

0  0 

0 

15  1.597 

58 

1 

1 

0 

0 

0 

0  0  0 

0  1 

1 

0  0 

0  0  0  0  0 

0  0  1 

0  0 

1 

1  0 

0 

26  1.362 

59 

0 

1 

0 

1 

0 

0  0  0 

0  1 

1 

0  0 

0  0  0  1  1 

0  0  0 

0  0 

0 

1  0 

0 

09  1.708 

60  ' 

0 

0 

0 

1 

0 

0  0  1 

0  0 

0 

0  1 

0  10  0  1 

10  1 

0  1 

0 

1  0 

0 

07  1.805 

Note; 

Items  1 -25  «e  from  section  2  and  items  26-60  are  from  sections 
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Table  1.2.3 

Items  Required  in  Each  of  the  27  Attributes 
Attribute  Items  (1-60) 


01  3.  4.  8,  9,  1 1.  16.  20.  26.  27.  23.  38.  39. 40.  41.  42.  44.  54.  56.  58 

02  1.  1 1.  19.  23.  25.  43.  50.  51.  54.  57.  58.  59 

03  13. 51 

04  2.  5.  10.  11.  13.  15.  19.  25.  33.  35.  36.  45.  47.  48.  59.  60 

05  2.  6.  13.  21.  29 

06  14.  17.  24.  30.  32.  37. 49.  52.  53.  55 

07  7.  10.  24.  34. 46.  52.  57 

08  18.  22. 60 

09  12.  25. 28. 35. 48.  53 

10  8.  1 1.  12. 16. 19.  20.  23. 25.  28.  30.  31.  32.  41. 42. 44.  50.  54.  55.  58.  59 

11  8.  1 1.  12. 16. 19.  20.  22. 23.  27. 28.  34. 41. 42. 47.  53.  54. 56.  58.  59 

12  1.  2.  5.  6.  9,  13.  15.  21. 26.  29.  31.  33.  35.  36. 40.  43. 44. 45. 48.  50.  51 

13  3,  4. 7. 10. 18. 24. 25.  30,  32,  37,  38,  39.  46.  57,  60 

14  14,  17. 49, 52, 55 

15  1,5.6,8.11.13.14,15,  18.  19.22,24.27.28,31,53,54.56.60 

16  33.  34.  35.  36.  37. 38, 39, 40, 41,  42, 43, 44,  45.  46. 47,  48,  49,  50,  51. 52 

17  14, 21,  31, 32,  34.  35. 41. 42. 45, 47, 48,  49,  55,  59 

18  1,  2,  3, 4,  5.  6,  9.  10, 11, 12, 13,  15,  19,  21,  22,  23.  24,  25,  26, 27,  29, 

36.  37. 40, 41, 42,  43, 46.  50,  53.  57, 59,  60 

19  10. 18. 24, 52, 57.  60 

20  7.  8.  1 1.  16,  18,  19,  20.  22,  23,  24.  25,  27.  28,  31.  51,  53,  56 

21  17, 23.  24,  25,  49.  50.  55,  56, 58,  60 

22  22.  33,  45,  54 

23  3.  7.  8.  14,  17,  20.  21.  22,  23,  30,  31,  32.  41,  42.  44,  47.  49.  54,  56,  60 

24  5,  7,  8,  20.  28.  29,  51,  54,  56,  58 

25  1.  3,  4,  10,  15,  16, 17,  18,  19.  20.  21,  22,  23,  24,  25,  39.  41,  42, 49,  50,  52, 

55,  56,  58,  59,  60 

26  3,  4,  38.  39,  56 

27  21, 31 


27 


Tabic  1.3.1 


Multiple  Regression  Results:  Predicting  Item  Difficulties  from  27  Attributes. 


Attribute 

SEb 

t 

A1 

.02 

6.61 

.01 

A2 

-19.54 

6.41 

-3.05*** 

A3 

-21.67 

11.36 

-1.91* 

A4 

-11.51 

5.47 

-2.10** 

A5 

-4.44 

8.84 

-.50 

A6 

-21.80 

10.10 

-2.16** 

A7 

-12.35 

11.25 

-1.10 

A8 

-29.39 

16.56 

-1.78* 

A9 

-10.46 

7.33 

-1.43 

AlO 

-2.78 

6.27 

-.44 

All 

-14.92 

12.10 

-1.23 

A12 

-8.11 

12.67 

-.64 

A13 

3.03 

10.55 

.29 

A14(.) 

0.00 

0.00 

0.00 

A15 

10.57 

5.51 

1.92* 

A16 

-7.90 

5.38 

-1.47 

A17 

-4.18 

6.38 

-.66 

A18 

-1.92 

5.51 

-.35 

A19 

-26.93 

10.84 

-2.48** 

A20 

-4.32 

6.01 

-.72 

A21 

-13.44 

6.28 

-2.14** 

A22 

-3.76 

8.51 

-.44 

A23 

-8.57 

5.10 

-1.68 

A24 

-8.45 

6.59 

-1.28 

A2S 

-16.71 

5.44 

-3.08*** 

A26 

-4.75 

11.72 

-.41 

A27 

-15.87 

13.86 

-1.15 

r2  =  0.83 

Note: 

A1  to  A27 :  Initial  set  of  attiibutj;s  (see  Table  1.2.1). 

Y :  Percent  of  cmiect  responses  (as  iq)orted  in  Taldng  the  SAT  1990-91). 
Number  of  items:  60  (1-25  from  Section  2;  26-60  from  Section  5). 

(-)  Parameter  not  estimated  (A14  is  a  linear  combination  of  All,  A12,  A13) 
•  p<.10.  ••  p<.05.  ***p<.01. 
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Table  1.3.2 


The  Reduced  set  of  IS  Attributes 


Attribute  Attribute's  Description 

No. 


A.  Content  related  attributes 

1.  Arithmetics  (including  content  of  attributes:  1, 4,  S,  22). 

2 .  Elementary  Algebra  (incluthng  content  oi  attributes:  2, 27). 

3.  Advanced  Algebra. 

6.  Elementary  Geometry  (including  content  of  attributes:  6, 7, 8, 26). 
1 1.  Word  problems. 

15.  Logic  (if...then). 

1 6.  Quantitative  comparisons*. 


B.  Process  Related  Attributes 

17.  Understanding  of  the  meaning  of  concepts. 

1 8.  Application  of  simple  rules/algorithms  (SOLVE:  perform  computations). 

1 9.  Cmr^hension  +  applicadtm  of  rules/dteorems  (chooses  and  applies 
conectly). 

20.  Reasoning  (creates  an  e^^uadon). 

21.  Analytic  thinking,  arrive  lestrucmiing  ^gher  mental  processes). 

23.  Reading  ctnnprehenrion  (4-  follow  instructions;  madi/geometry  terminolo^). 

24.  Test-wiseness  (solves  intuitively;  by  example;  goes  b^kwards  from  the  given 
answers). 

25.  Number  of  steps  in  the  solution  >  1 


*  Applies  to  section  5  cmly. 
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Table  1.3.3 

The  25  items  ci  section  2  Listed  by  the  Reduced  set  of  14  Attributes 


hem 

Attribute 

01 

2,  15.  18.  25 

02 

1.  18 

03 

1. 6. 18. 23. 25 

04 

1.  6. 18.  25 

05 

1.  15.  18.  24 

06 

1.  15.  18 

07 

6.  20.  23.  24 

08 

1.  11.  15.  20.  23.24 

09 

1. 18 

10 

1. 6. 18. 19. 25 

11 

1.2.11.15.18.20 

12 

11. 18 

13 

1.3.15.18 

14 

6.15.  17.23 

15 

1. 15. 18.25 

16 

1.11.20.25 

17 

6.  21.  23. 25  ■ 

18 

6.15,19.20.25 

19 

1,2,11,  15.18.20,25 

20 

1, 11,  20. 23, 24,  25 

21 

1.  2,  17.  18. 23,  25 

22 

1.  6. 11.  15. 18. 20.  23. 25 

23 

2.  11.  18.20,21.  23,25 

24 

6, 15. 18. 19. 20.  21, 25 

( 

25 

1,2, 18,20.21,25 

i 

:1 
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Tabic  1.4.1 

Incidence  Matrix  Q  for  14  Attributes  by  25  Items  and  the  Item  Parameters 
(IRT:  a's.  b*s)  and  Percent  Correct 


Item  1111122222  IRT  % 

No.  1  2  3  6  1  5  7  8  9  0  1  3  4  5  a*s  b's  Correct 


1 

0 

1  0 

0 

0 

1 

0 

1 

0 

0 

0 

0 

0 

1 

.685 

-1.518 

81 

2 

1 

0  0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

.833 

-1.938 

89 

3 

1 

0  0 

1 

0 

0 

0 

1 

0 

0 

0 

1 

0 

1 

1.089 

-1.499 

88 

4 

1 

0  0 

1 

0 

0 

0 

1 

0 

0 

0 

0 

0 

1 

.593 

-1.285 

76 

5 

1 

0  0 

0 

0 

1 

0 

1 

0 

0 

0 

0 

1 

0 

.855 

-1.298 

80 

6 

1 

0  0 

□ 

0 

1 

0 

1 

0 

0 

0 

0 

0 

0 

1.274 

-1.309 

85 

7 

0 

0  0 

1 

0 

0 

0 

0 

0 

1 

0 

1 

1 

0 

.263 

-2.180 

72 

8 

1 

0  0 

□ 

1 

1 

0 

0 

0 

1 

0 

1 

1 

0 

.491 

-.580 

59 

9 

1 

0  0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

-1.036 

76 

10 

1 

0  0 

1 

0 

0 

0 

1 

1 

0 

0 

0 

0 

1 

.869 

-.066 

53 

11 

1 

1  0 

0 

1 

1 

0 

1 

0 

1 

0 

0 

0 

0 

.855 

-.480 

60 

12 

0 

0  0 

0 

1 

0 

0 

1 

0 

0 

0 

0 

0 

0 

.692 

-.566 

64 

13 

1 

0  1 

0 

0 

1 

0 

1 

0 

0 

0 

0 

0 

0 

.847 

-.452 

58 

14 

0 

0  0 

1 

0 

1 

1 

0 

0 

0 

0 

1 

0 

0 

.677 

-1.134 

72 

IS 

1 

0  0 

0 

0 

1 

0 

1 

0 

0 

0 

0 

0 

1 

.510 

.039 

48 

16 

1 

0  0 

0 

1 

0 

0 

0 

0 

1 

0 

0 

0 

1 

.593 

-.245 

55 

17 

0 

0  0 

1 

0 

0 

0 

0 

0 

0 

1 

1 

0 

1 

.725 

.422 

38 

18 

0 

0  0  1 

0“  1 

0 

0 

1 

1 

0 

0 

0 

1 

.614 

.645 

35- 

19 

1 

10  0 

1 

1 

0 

1 

0 

1 

0 

0 

0 

1 

.664 

.694 

33 

20 

1 

0  0  0 

1 

0 

0 

0 

0 

1 

0 

1 

1 

1 

.574 

.772 

32 

21 

1 

1  0 

0 

0 

0 

1 

1 

0 

0 

0 

1 

0 

1 

.858 

.860 

27 

22 

1 

0  0 

1 

1 

1 

0 

1 

0 

1 

0 

1 

0 

1 

.993 

.987 

23 

23 

0 

10  0 

1 

0 

0 

1 

0 

1 

1 

1 

0 

1 

.464 

2.035 

19 

24 

0 

0  0 

1 

0 

1 

0 

1 

1 

1 

1 

0 

0 

1 

.501 

2.333 

14 

25 

1 

1  0 

0 

0 

0 

0 

1 

0 

1 

1 

0 

0 

1 

.536 

1.944 

17 

31 


Table  1.4.2 


Multiple  Regression  Results:  Predicting  Item  Difficulties  for  Items  1-25  from  14 

Attributes. 


Proportion  Correct 

IRT  b- values 

Attribuie 

b 

SEb 

P 

t 

b 

SEb 

P 

t 

25 

-16.06 

8.53 

-.33 

-1.88 

.85 

.31 

.35 

2.72* 

23 

-4.38 

12.85 

-.09 

-.43 

.21 

.47 

.08 

.45 

21 

-36.60 

12.52 

-.57 

-2.92* 

2.35 

.46 

.70 

5.10** 

11 

-13.84 

11.94 

-.27 

-1.16 

.84 

.44 

.32 

1.92 

03 

-21.46 

17.77 

-.18 

-1.21 

.75 

.65 

.12 

1.14 

IS 

-2.81 

7.24 

-.06 

-.39 

.26 

.27 

.11 

.98 

02 

3.69 

12.89 

.07 

.29 

-.42 

.47 

-.15 

-.89 

01 

-5.29 

9.45 

-.10 

-.56 

.36 

.35 

.14 

1.05 

24 

-.29 

13.59 

-.00 

1 

« 

o 

-.16 

.50 

-.05 

-.32 

19 

-19.39 

15.42 

-.27 

-1.26 

.1*26 

.57 

.33 

2.22 

18 

-1.44 

11.56 

-.03 

-.13 

.23 

.43 

.09 

.55 

17 

-26.97 

17.40 

-.31 

-1.55 

1.35 

.64 

.30 

2.10 

06 

6.61 

12.48 

.11 

.45 

-.57 

.46 

-.22 

■1.24 

20 

-12.54 

13.43 

-.26 

-.93 

.37 

.49 

.15 

.76 

a 

89.00 

12.59 

-2.06 

.46 

R2 

R^adj. 

.83 

.59 

.91 

.79 

•p<.05  ;  •*p<.001 
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T*bl*  3.1  A  list  of  Cognitive  States  in  which  at  least  Five 
Percent  of  Examinees  are  Classified  (N  -  2334} 

Cognitive  Frequency  Attribute  Mastery  Attributes  not 
States  Pattern  mastered 


1111122222 

12361578901345 

1 

19 

11111111111111 

2 

180 

11111111110111 

21 

4 

32 

11111101111111 

17 

5 

18 

11011111111111 

3 

6 

94 

11011111110111 

3.  21 

8 

11 

11011101110111 

3.17.  21 

9 

37 

11111111011111 

19 

10 

46 

11111111010111 

19.  21 

12 

28 

11111101010111 

17,19,21 

14 

30 

11011111010111 

3.19.21 

22 

18 

11010111110111 

3.11,21 

28 

12 

11110101010111 

11,17,19,21 

30 

38 

11010111010111 

3.11,19,21 

34 

17 

11111111100111 

20,21 

66 

87 

11111111110101 

21,24 

70 

11 

11011111110101 

21,24 

126 

25 

11111101110011 

21,23 

128 

14 

11011101110011 

3,17,21,23 

138 

40 

11111101010011 

17,19,21,23 

140 

16 

11011101010011 

3,17,19,21,23 

215 

12 

01011111111111 

1.3 

217 

30 

OlOlllllllOlll 

1.3,21 

218 

14 

01011111010111 

1,3,19,21 

220 

13 

01011101011111 

1.3,17,19 

221 

24 

01011101110111 

1.3,17,21 

222 

22 

01011101010111 

1,3,17,19,21 

253 

43 

lOllllllllOlll 

2,21 

257 

15 

10011111110111 

2,3,21 

261 

30 

10111111010111 

2,19,21 

268 

11 

10110111111111 

2,11 

269 

31 

10110111110111 

2,11,21 

273 

12 

10010111110111 

2,3,11,21 

277 

13 

10110111010111 

2,11,19,21 

468 

67 

11111111010110 

19,21,25 

469 

18 

11111101010110 

17,19,21,25 

472 

132 

10111111010110 

2,19,21,25 

473 

25 

10111101010110 

2.17,19,21,25 

474 

32 

10011111010110 

2,3,19,21,25 

475 

15 

10011101010110 

2.3.17,19,21,25 

476 

39 

10110111010110 

2,11,19,21,25 

477 

16 

10110101010110 

2,11,17,19,21,25 

478 

36 

10010111010110 

2,3,11,19,21,25 

488 

11 

10111111000110 

2,19,20,21,25 

502 

16 

11111111010100 

19,21,24,25 

520 

33 

10011001010110 

2,3,15,17.19.21,25 

547 

11 

00011111111111 

1.2.3 
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Table  3.2  Ability  Levels  and  Atypicality  of  Cognitive 
States  (sorted  by  B  values) 


Cognitive  Frequency  6  C  Attributes  not 

States  mastered 


1 

19 

5 

18 

9 

37 

2 

180 

6 

94 

10 

46 

4 

32 

66 

87 

14 

30 

8 

11 

12 

28 

70 

11 

253 

43 

257 

15 

261 

30 

126 

25 

34 

17 

268 

11 

22 

18 

128 

14 

269 

31 

138 

40 

30 

38 

273 

12 

140 

16 

28 

12 

277 

13 

468 

67 

469 

18 

472 

132 

473 

25 

488 

11 

502 

16 

474 

32 

476 

39 

257 

15 

477 

16 

478 

36 

215 

12 

547 

11 

217 

30 

220 

13 

520 

33 

218 

14 

221 

24 

222 

22 

5.00 

0.52 

3.06 

1.43 

1.98 

0.67 

1.83 

-1.37 

1.40 

-0.59 

1.14 

-0.69 

1.12 

-0.55 

0.88 

0.16 

0.83 

-0.01 

0.82 

0.14 

0.61 

-0.22 

0.60 

0.94 

0.59 

0.06 

0.34 

0.55 

0.15 

-0.06 

0.04 

0.52 

0.01 

-0  74 

•0.02 

0./9 

-0.17 

-0.26 

•0.19 

0.88 

-0.35 

-0.51 

-0.35 

0.19 

-0.54 

-1.18 

-0.56 

-0.54 

-0.57 

0.26 

-0.71 

-1.58 

-0.71 

-1.37 

-0.75 

-0.45 

-0.91 

-0.10 

-1.11 

-0.33 

-1.12 

-0.32 

-1.13 

0.07 

-1.13 

0.70 

-1.16 

-0.87 

-1.24 

-0.64 

-1.32 

-0.65 

-1.40 

-0.40 

-1.45 

-1.00 

-1.49 

2.32 

-1.81 

1.82 

-1.99 

-0.11 

-2.02 

0.66 

-2.07 

-0.69 

-2.22 

-0.82 

-2.25 

-0.18 

-2.55 

-0.90 

19 

21 

3.21 

19.21 
17 

21,24 

3.19.21 

3.17.21 

17.19.21 
21,24 

2.21 

2.3.21 

2.19.21 

21.23 

20.21 
2.11 

3.11.21 

3.17.21.23 

2.11.21 

17.19.21.23 

3.11.19.21 

2.3.11.21 

3.17.19.21.23 

11.17.19.21 

2.11.19.21 

19.21.25 

17.19.21.25 

2.19.21.25 

2.17.19.21.25 

2.19.20.21.25 

9.21.24.25 

2.3.19.21.25 

2.11.19.21.25 

2.3.17.19.21.25 

2.11.17.19.21.25 

2.3.11.19.21.25 

1.3 

1.2.3 

1.3.21 
1.3,17.19 

2.3.15.17.19.21.25 

1.3.19.21 

1.3.17.21 

1.3.17.19.21 
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Table  3.3  Descriptive  Scetistics  of  the  14  Attributes  and  0,  C 
and  Generalized  Zs  (N  -  2334) 


Attributes 

mean 

S.D.  Corr. 

with  0 

Corr.  with  C 

1 

.896 

.305 

.25 

-.11 

2 

.631 

.483 

.21 

-.03 

3 

.542 

.498 

.30 

-.17 

6 

.958 

.201 

.16 

-.25 

11 

.764 

.425 

.21 

-.01 

15 

.939 

.240 

.19 

-.12 

17 

.668 

.471 

.25 

-.18 

18 

.978 

.152 

.15 

-.14 

19 

.461 

.499 

.22 

.14 

20 

.879 

.326 

.19 

-.01 

21 

.213 

.409 

.11 

.58 

23 

.901 

.298 

.05 

-.11 

24 

.807 

.395 

.17 

-.34 

25 

.790 

.408 

.15 

.27 

Dimension 

mean 

S.D. 

e 

.060 

1.200 

c 

-.147 

1.067 

Cl 

-.089 

1.002 

C2 

-.050 

.992 

C3 

-.055 

1.028 

C4 

-  .076 

1.010 

<5 

-.027 

1.008 
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Table  4.3.1  Examples  of  Probablllcy  Vectors  for  the  First  Ten 
Examinees 


14  Attributes 


1  500  93  65  59  98  82  96  74  99  47  89  15  92  82  77 

2  640  98  75  73  97  84  99  80  100  58  96  23  90  86  86 

3  420  89  59  49  97  75  94  71  97  41  85  13  92  82  76 

4  510  94  65  60  98  82  96  75  99  47  89  14  91  82  77 

5  730  100  82  79  100  87  100  75  100  66  100  37  91  89  95 

6  340  80  53  36  93  64  89  49  94  36  81  26  88  74  75 

7  790  100  86  83  100  99  100  100  100  72  100  79  94  99  100 

8  230  53  44  17  71  46  66  21  90  28  75  32  78  52  68 
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Table  5.1 


Form  OA;  Incidence  Matrix  Q  for  14  Attributes  and  25  Items 


Item  1111122222  % 

No.  12361578901345  Correct 


01 

0 

1 

0 

0 

0 

1 

0 

1 

02 

1 

0 

0 

0 

1 

1 

1 

0 

03 

1 

0 

0 

0 

1 

1 

0 

1 

04 

1 

1 

0 

0 

0 

1 

0 

1 

05 

0 

1 

0 

0 

1 

0 

1 

1 

06 

1 

0 

0 

0 

1 

1 

1 

1 

07 

0 

0 

0 

1 

1 

0 

1 

0 

08 

1 

0 

0 

0 

0 

1 

0 

1 

09 

0 

0 

0 

1 

1 

0 

1 

0 

10 

1 

1 

0 

0 

1 

0 

0 

1 

11 

0 

0 

0 

1 

1 

0 

0 

0 

12 

1 

0 

0 

0 

1 

0 

0 

0 

13 

0 

1 

1 

0 

1 

1 

0 

1 

14 

0 

0 

1 

1 

0 

0 

1 

1 

15 

0 

0 

0 

1 

1 

0 

1 

0 

16 

0 

0 

0 

1 

0 

1 

1 

0 

17 

1 

1 

1 

0 

0 

1 

0 

1 

18 

1 

0 

0 

0 

1 

0 

1 

0 

19 

0 

1 

1 

1 

1 

0 

0 

1 

20 

0 

0 

1 

0 

0 

1 

1 

0 

21 

0 

0 

1 

0 

0 

1 

1 

1 

22 

0 

0 

0 

1 

0 

0 

1 

1 

23 

1 

0 

0 

0 

1 

1 

0 

0 

24 

0 

0 

1 

0 

0 

1 

0 

1 

25 

1 

0 

0 

0 

1 

0 

1 

1 

000000  89.2 

000000  89.0 
000010  78.6 

100010  70.6 

010001  73.9 

010000  54.9 

000100  73.7 

000000  79.5 

000100  73.6 
000001  63.1 

100100  68.1 

000111  83.4 

010000  51.8 

101101  25.7 

011110  46.1 

001000  59.6 

000001  34.5 

0  0  1  0  0  1  29^5 
101001  21.8 
0  0  1  1  0  0  51.0 
001001  22.4 

011001  26.2 
111111  20.3 
100001  25.1 

111101  19.8 
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Tabic  5.2 

Multiple  Regression  Results:  Predicting  Item  Difficulties  of  25  Items 


(Form  OA  Section  2)  From  14  Attributes. 


Attribute 

b 

SEb 

P 

t 

25 

-14.60 

9.00 

-.31 

-1.62 

11 

.85 

8.36 

.02 

.10 

17 

3.10 

7.91 

.07 

.39 

06 

-9.80 

9.78 

-.19 

-1.00 

24 

6.43 

7.86 

.11 

.82 

20 

-17.38 

7.97 

-.33 

-2.18 

19 

-10.69 

7.05 

-.20 

-1.52 

03 

-23.12 

9.41 

-.44 

-2.46* 

02 

8.42 

7.69 

.16 

1.10 

23 

6.57 

9.52 

.13 

.69 

21 

-19.19 

7.68 

o 

1 

-2.50* 

01 

-10.27 

8.05 

-.21 

-1.28 

18 

1.65 

9.02 

.03 

.18 

15 

-1.29 

12.67 

-.03 

-.10 

a 

81.47 

19.97 

R2 

.91 

R^adj. 

• 

-4 

00 

*  p<X)5  **  p<  .001 
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Tabic  5.3 

Incidence  Matrix  Q  for  15  Attributes  and  25  Items  (Form  OA  section  2)  and  Percent 

Qvxect 


Item 

T 

T 

T 

T 

1  2  2  2  2  n 

- % 

No. 

1 

2 

3 

6 

1 

5 

7 

8 

9 

0 

1 

3 

4 

5 

6 

Conect 

01 

0 

1 

0 

0 

0 

1 

0 

1 

0 

0 

0 

0 

0 

0 

0 

89.2 

02 

1 

0 

0 

0 

1 

1 

1 

0 

0 

0 

0 

0 

0 

0 

0 

89.0 

03 

1 

0 

0 

0 

1 

1 

0 

1 

0 

0 

0 

0 

1 

0 

0 

78.6 

04 

1 

1 

0 

0 

0 

1 

0 

1 

1 

0 

0 

0 

1 

0 

0 

70.6 

05 

0 

1 

0 

0 

1 

0 

1 

1 

0 

1 

0 

0 

0 

1 

0 

73.9 

06 

1 

0 

0 

0 

1 

1 

1 

1 

0 

1 

0 

0 

0 

0 

0 

54.9 

07 

0 

0 

0 

1 

1 

0 

1 

0 

0 

0 

0 

1 

0 

0 

0 

73.7 

08 

1 

0 

0 

0 

0 

1 

0 

1 

0 

0 

0 

0 

0 

0 

0 

79.5 

09 

0 

0 

0 

1 

1 

0 

1 

0 

0 

0 

0 

1 

0 

0 

0 

73.6 

10 

1 

1 

0 

0 

1 

0 

0 

1 

0 

0 

0 

0 

0 

1 

1 

63.1 

11 

0 

0 

0 

1 

1 

0 

0 

0 

1 

0 

0 

1 

0 

0 

0 

68.1 

12 

1 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

1 

1 

1 

0 

83.4 

13 

0 

1 

1 

0 

1 

1 

0 

1 

0 

1 

0 

0 

0 

0 

0 

51.8 

14 

0 

0 

1 

1 

0 

0 

1 

1 

1 

0 

1 

1 

0 

1 

0 

25.7 

15 

0 

0 

0 

1 

1 

0 

1 

0 

0 

1 

1 

1 

1 

0 

0 

46.1 

16 

0 

0 

0 

1 

0 

1 

1 

0 

0 

0 

1 

0-0 

0 

0 

59.6 

17 

1 

1 

1 

0 

0 

1 

0 

1 

0 

0 

0 

0 

0 

1 

0 

34.5 

18 

1 

0 

0 

0 

1 

0 

1 

0 

0 

0 

1 

0 

0 

1 

1 

29.5 

19 

0 

1 

1 

1 

1 

0 

0 

1 

1 

0 

1 

0 

0 

1 

0 

21.8 

20 

0 

0 

1 

0 

0 

1 

1 

0 

0 

0 

1 

1 

0 

0 

0 

51.0 

21 

0 

0 

1 

0 

0 

1 

1 

1 

0 

0 

1 

0 

0 

1 

0 

22.4 

22 

0 

0 

0 

1 

0 

0 

1 

1 

0 

1 

1 

0 

0 

1 

0 

26.2 

23 

1 

0 

0 

0 

1 

1 

0 

0 

1 

1 

1 

1 

1 

1 

0 

20.3 

24 

0 

0 

1 

0 

0 

1 

0 

1 

1 

0 

0 

0 

0 

1 

0 

25.1 

25 

1 

0 

0 

0 

1 

0 

1 

1 

1 

1 

1 

1 

0 

1 

0 

19.8 
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Table  5.4 


The  25  Items  (Form  OA  Section  2)  Listed  by  the  15  Attributes 


Item 

Attribute 

Item  Attribute 

01 

2,  15,  18 

02 

1,  11,  15,  17 

03 

1,  11,  15,  18,  24 

04 

1,  2,  15,  18,  19,  24 

05 

2,  11,  17,  18,  20,  25 

06 

1,  11,  15,  17,  18,  20,  07,  6,  11,  17,  23 

08 

1,  15,  18 

09 

6,  11,  17,  23 

10 

1,  2,  11,  18,  25,  26* 

11 

6,  11,  19,  23 

12 

1,  11,  23,  24,  25 

13 

2,  3,  11,  15,  18,  20 

14 

3,  6, 17,  18,  19,  21,  23,  25 

15 

6,  11,  17,  20,  21,  23,  24 

16 

6, 15,  17,  21 

17 

1,  2,  3,  15,  18,  25 

18 

1,  11,  17,  21,  25,  26 

19 

2,  3,  6,  11,  18,  19,  21,  25 

20 

3,  15,  17,  21,  23 

21 

3,  15, 17,  18,  21,  25 

22 

6,  17,  18,  20,  21,  25 

23 

1,  11,  15,  19,  20,  21,  23,  24,  25 

24 

3,  15,  18,  19,  25 

25 

1,  11,  17,  18,  19,  20,  21,  23,  25 
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Table  5.5 


The  15  Attributes  Listed  by  the  Items  in  which  They  Are  Required 


Attribute 


Items  (1-25  form  OA  section  2) 


1 

2 

3 

6 

11 

15 

17 

18 

19 

20 
21 

23 

24 

25 

26 


2.  3. 4.  6.  8.  10.  12,  17,  18,  23,  25 

1,  4,  5,  10,  13. 17.  19 
17,  19. 20. 21. 24 

7.  9.  11.  14. 15. 16,  19,  22 

2.  3.  5,  6, 7.  9. 10, 11, 12, 13,  15, 18, 19,  23,  25 

1,  2,  3. 4. 6.  8.  13, 16,  17,  20.  21. 23.  24 

2,  3,  5. 6.  7.  14,  15,  16.  18,  20.  21, 22.  25 
1,  3.  4. 5.  8,  10,  13. 17, 19. 22.  24,  25 
12. 14. 19. 25 

10.  13. 15. 22,  23,  25 
12, 16. 18. 19. 20,  21. 23 
3,7.9,11. 12, 14,15,23^25 
3  4, 12. 15,23 

5,  10, 12, 14, 17,  18,  19,  21,  22, 24. 25 
1,  18 


41 


Table  5.6 


Multiple  Regression  Results:  Predicting  Item  Difficulties  (percent  ccMrect)  fen*  Items  1-25 

(Form  OA  Section  2)  From  IS  Attributes. 


Attribute 

b 

SEb 

P 

t 

26 

-26.35 

12.57 

-.30 

-2.10 

17 

-.49 

7.05 

-.01 

-.07 

03 

-23.53 

8.13 

-.44 

-2.89* 

23 

-.53 

8.89 

-.01 

-.06 

20 

-20.72 

7.06 

-.39 

-2.93* 

06 

-17.82 

9.28 

-.35 

-1.92 

19 

-12.38 

6.15 

-.23 

-2.01 

24 

2.04 

7.10 

.03 

.29 

11 

1.22 

7.23 

.03 

.17 

25 

-16.94 

7.86 

VO 

• 

1 

-2.16 

02 

9.23 

6.65 

.17 

1.39 

18 

-2.97 

8.10 

-.06 

-.37 

01 

-6.61 

7.17 

-.14 

-.92 

21 

-12.63 

7.33 

-.26 

-1.72 

15 

-13.27 

12.35 

-.28 

-1.07 

a 

98.41 

19.06 

r2 

.94 

R^adj. 

.83 

•  p<.05 
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Figure  4.4.1  Response  Function  for  Attribute  19,  SAT  Hathenatics 
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Figure  4.5.2  A  Prototype  Student  Report  2 
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Response  Function  for  Attribute  19,  SAT-M 


SAT-M  Scale 


Attribute  19  represents  the  ability  to  comprehend  and  apply  rules  and  the<mms  correctly. 

The  figure  shows  an  enqiirical  attribute  response  function  (points  denoted  by  x)  and 
posterior  median  estimates  of  selected  values  of  the  corre^xinding  theoretical  fimction 
(cmmected  by  straight  lines). 

The  posterior  mean  estimate  of  the  25%  point  fen-  this  function  is  277. 

The  posterior  mean  estimate  of  the  50%  point  for  this  functicHi  is  546. 

The  postericH*  mean  estimate  of  the  75%  point  for  this  functicMi  is  760. 
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1)  A  student  report,  Kuai  Tetsuoke 


SAT  percentile  score  based  on  i tea* level:  60 
Your  percentile  scores  on  the  content  area  are 


Arithmetic . A 

Algebra . C 

Geometry . C 

Miscellaneous . D 


Performance  xmderlylng  cognitive  processes: 

Understanding  the  meaning  of  concepts . C 

Application  of  simple  rules/algor Ithms 
(solving  equations,  computation,  derivation 
of  simple  algebraic  expressions) . A 

Comprehension  and  application  of 
rules/theoresis ,  principles  correctly . C 

Reading  comprehension  (-t-follov 
instructions ; math/geometry  terminology) . . .  B 

Reasoning  (create  an  equation,  identifying 
components  and  follov  procedures) . C 

Analytic  thinking,  cognitive  restructurelng 
(hl^er  smntal  processes) . D 

Strategies  (trial* and- errors  by  plug  in 
numbers,  make  an  Inference  of  the  correct 
answer  from  options  with  unknown  systematic 


methods .  C 

The  complex  problems  with  steps  >  1 . C 


A:  top  10  percent 
B:  70  -  89  percentile 
C:  Average 

D:  30  -  49  percentile 
E:  10  -  29  percentile 
F:  bottom  10  percent 


2)  A  student  report  for  Jane  Smith 


SAT-scaled  score  based  on  item  performance:  600 

Probed)ility  of  success  on  attribute (s)  associated  with: 

Mean  at  Your 
600-level  score 


Arithmetic . 97  %  ok 

Elementary  Algebra .  72  %  no 

Advanced  algebra .  69  %  no 

Geometry . 97  %  ok 

Understanding  the  meaning  of  concepts .  76  %  ok 

Application  of  simple  rules/algorithms 
(solving  equations,  computation,  derivation 
of  simple  algebraic  expressions) .  100  %  ok 

Comprehension  and  application  of 

rules/ theorems ,  principles  correctly .  54  %  no 

Reading  comprehension  (tfollov 

instructions; math/geometry  terminology) .  90  %  no 

Reasoning  (create  an  equation,  identifying 

components  and  follow  procedures) .  94  %  no 

Analytic  thinking,  cognitive  restructureing 
(higher  mental  processes) .  17  %  ok 

Strategies  ( trial -and-errors  by  plug  in 

numbers,  make  an  inference  of  the  correct 

answer  from  options  with  \inknown  systematic 

methods .  85  %  ok 


Mastery  of  complex  problems  with  steps  >  l..  82  %  ok 

Additional  Comments: 

Your  performance  pattern  is  rather  xinusual,  so  we  provide  you 
with  your  diagnosed  cognitive  state  on  the  right  most  side  of 
the  above  table. 

we  recommend  that  you  practice  word  problems  and  pay  more 
attention  to  the  meaning  of  principles,  theorems  and 
properties . 

You  should  also  follow  the  instructions  more  carefully. 
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XI.  A  report  for  «  class  rooa  tsachsr 

Class  size  10,  five  girls  and  five  boys 
junior  year,  Teacher  is  Mrs  Saith 

The  aean  of  SAT* scale  score:  450 
The  standard  deviation  30 


naaes 

SAT 

percentile 

attributes 

scale 

rank 

Al 

A2 

A3 

A4 

A5 

A6 

A7 

A8 

A9 

1. 

Donald  Duck 

750 

95% 

90 

85 

50 

85 

60 

77 

81 

92 

65 

2. 

Wylie  Cayote 

540 

61% 

81 

64 

55 

42 

89 

45 

32 

75 

18 

3. 

Mickey  Motise 

605 

80% 

82 

71 

62 

40 

80 

55 

54 

67 

32 

4. 

Olive  Oyl 

680 

90% 

88 

67 

32 

97 

65 

46 

98 

63 

88 

5. 

Bo  Peep 

e  •  •  •  < 

442 

B  • 

67% 

43 

S3 

65 

24 

35 

36 

56 

46 

67 

10 

.  Charlie  Broim  590 

69% 

75 

60 

SO 

40 

85 

46 

42 

77 

29 

Average 

620 

74% 

76 

72 

65 

54 

67 

51 

46 

43 

25 

S.D. 

42 

5 

7 

8 

10 

11 

9 

15 

12 

17 
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APPENDIX 
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Appendix  I 


The  rule-space-model  has  recently  been  introduced  in  various  ETS 
technical  reports  (Tatsuoka  &  Tatsuoka,  1992;  Sheehan,  Tatsuoka  &  Lewis,  1993; 
Birenbavun  &  Tatsuoka.  1993;  Birenbaum,  Kelly  &  Tatsuoka,  1993).  This  paper 
emphasizes  the  introduction  of  the  procedures  that  lead  to  probability 
functions  for  attributes  (PFAs) ,  which  are  applied  to  SAT  Mathematics  tests. 

An  PFA  is  the  conditional  probability  function  for  successful  performance  on 
each  attribute  at  given  IRT  ability  level  0, 

PAk(e)  -  Prob(  Ak  -  II  0),  k  -  1.  2 . K  (1) 

Since  PFAs  are  defined  on  the  IRT  ability  variable  0  or  equivalently,  on  the 
SAT  scale  that  are  obtained  by  transforming  the  0-scale,  each  scale  point  is 
associated  with  a  probability  vector  of  the  cognitive  attributes. 

1.  An  Incidence  Matrix  and  All  Possible  Ideal-Item-Score  Patterns 

Tatsuoka  (1990)  organized  the  underlying  cognitive  tasks  that  are 
required  in  answering  test  items  in  an  incidence  matrix,  Q-matrix,  whose  rows 
represent  attributes  (i.e.,  knowledge,  cognitive  processes  and  skills  etc.) 
and  columns  represent  items.  The  entries  in  each  column  indicate  which 
attributes  are  involved  in  the  solution  of  each  item.  The  incidence  matrix  of 
order  that  relates  the  25  items  in  Section  2  of  the  SAT  M  with  the  14 
attributes  selected  in  the  previous  section  is  used  for  deriving  all  possible 
ideal-item-score  patterns  which  correspond  to  attribute  mastery  patterns 
(Tatsuoka,  1991).  The  expression  "ideal-item-score  patterns"  will  be  used 
hereafter  to  refer  to  logically  determined  knowledge  states,  as  contrasted 

•i 

with  the  examinees'  actual  item-response  patterns.  The  logically  determined 
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ideal-item-score  patterns  also  represent  classification  groups,  which 
correspond  to  the  attribute  mastery  patterns.  The  ideal-item-score  patterns 
are  the  images  of  a  Boolean  Descriptive  Function  (BDF)  that  is  defined  on  the 
lattice  of  attributes.  The  BDF  takes  the  value  of  either  one  or  zero,  for 
right  or  wrong  on  the  items.  The  definition  of  the  BDF  can  be  stated  by 
hypothesizing  that  "if  Attribute  Aj;  cannot  be  done  correctly"  or  equivalently 
"if  ^  is  not  mastered"  then  the  items  involving  Ajj  cannot  be  answered 
correctly.  The  value  of  one  for  Aj.  means  that  "one  can  do  Aj^  correctly"  which 
is  equivalent  to  "mastery  of  Ajj"  (Tatsuoka,  1991). 

An  algorithm  that  was  developed  by  Varadi  &  Tatsuoka,  1989  produces  all 
possible  ideal-item-score  patterns  from  an  incidence  matrix.  An  intuitive 
illustration  is  given  by  Tatsuoka  (1993) .  A  computer  program  BUGLIB  (Varadi  & 
Tatsuoka,  1989)  produced  more  than  3000  ideal— item— score  patterns  for  the 
incidence  matrix  of  order  27  x  60  in  Table  1.2.3,  and  600  for  that  of  order  14 
X  25  associated  with  Table  1.3.3.  Since  the  current  form  of  BUGLIB  cannot 
further  analyze  data  from  more  than  2000  groups,  the  discussion  in  this  report 
is  restricted  to  the  analysis  results  from  Table  1.3.3,  which  relates  to 
Section  2  of  SAT  M,  Form  8A.  Table  A.l  shows  a  partial  list  of  the  600  ideal— 
item-score  patterns . 

Insert  Table  A.l  about  here 

The  first  25  columns  after  the  IDs  give  the  ideal— item— score  patterns, 
followed  by  the  two  columns  showing  the  values  of  0  estimated  by  the  Maximum 
Likelihood  Methou  and  f  (Tatsuoka,  1984,  1985;  Tatsuoka  &  Linn,  1983),  and  the 
last  14  columns  show  the  corresponding  attribute  patterns.  The  m-th  ideal- 
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item-score  pattern  is  the  image  of  the  m-th  attribute  pattern  by  the  BDF.  The 
variable  f  will  be  described  in  Section  2.3. 

2.  A  set  of  "fuzzy"  response  patterns  There  are  2^*  possible  attribute 
patterns  for  14  attributes,  but  the  BDF  reduces  the  number  of  reliable 
attribute  patterns  to  600.  These  600  attribute  patterns  correspond  to  600 
ideal— item— score  patterns.  Conceptually,  an  item-response  pattern  that  does 
not  correspond  to  one  of  these  600  ideal— item-score  patterns  is  considered  to 
be  a  "fuzzy  item  patterns"  produced  by  slips.  Slips  are  regarded  as  deviations 
from  an  ideal-item-score  pattern. 

Bayes'  decision  rules  for  minimum  error  are  known  to  produce  optimal 
classification  and  are  also  known  to  be  relatively  unaffected  by  the 
distribution  of  scores  in  a  group.  Application  of  Bayes'  decision  rules  to  our 
classification  problem  requires  that  the  distribution  of  each  cognitive  state 
should  be  obtained  statistically. 

Tatsuoka  &  Tatsuoka  (1987)  introduced  a  slippage  random  variable  and 
slippage  probabilities  for  the  items,  and  explained  fuzzy  response  patterns  as 
outcomes  of  inconsistent  performance.  The  fuzzy  response  patterns  around  each 
ideal-item  score  pattern  will  cluster  together.  They  showed  that  a  set  of 
fuzzy  response  patterns  around  an  ideal— item-score  pattern  follows  a  compound 
binomial  distribution  with  slippage  probabilities  for  each  item.  Falmagne 
(1989)  formulated  a  model  that  estimates  these  slippage  probabilities. 

However,  if  the  number  of  cognitive  states  is  as  large  as  600,  we  would 
need  an  enormously  large  sample  for  estimating  the  parameters  of  the  model 
such  as  latent  class  models.  An  efficient  algorithm  for  estimating  very  large 
numbers  of  state  parameters  has  not  been  developed  yet.  The  rule  space  model 
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does  not  require  the  estimation  of  state  parameters  because  it  is  an 
analytical  approach,  and  the  probabilities  of  state  membership  for  an 
individual  will  be  obtained  through  a  classification  procedure. 

3.  Classification  space  and  bug  distributions  The  rule-space  model  takes  a 
statistical  pattern  classification  approach  to  achieve  classification  of 
examinees  into  one  of  600  cognitive  states.  An  advantage  of  this  approach  is 
that  the  problem  of  combinatorial  explosion  is  treated  geometrically  by 
mapping  all  patterns  —  both  the  examinees'  response  patterns  and  ideal-item- 
score  patterns  —  into  a  vector  space  in  which  an  appropriate  distance  is 
defined.  Moreover,  the  dimension  of  the  classification  space  usually  equals 
the  number  of  groups,  in  our  context  the  number  of  cognitive  states,  but  the 
model  reduces  the  dimension  of,  say  600,  to  as  few  as  two  dimensions.  If  two 
states  are  similar  in  terms  of  mastery  of  the  attributes,  they  are  located 
close  to  each  other  in  the  rule  space. 

The  vector  space  is  a  Cartesian  Product  space  of  0  and  the  image  of  a 
mapping  function  f(x,  0)  defined  by  Equation  2.3.1, 
f(X,  0)  =  (Pj(0)  -X,  Pj(0)  -  T(0)) 

=  b^Xi  +  b2X2  +  .  .  .  +  bjjXn  +  constant.  (2) 

Since  this  function  is  continuous,  the  fuzzy  response  patterns  around  a  given 
ideal-item-score  pattern,  R,  will  be  mapped  onto  points  in  the  vicinity  of  the 
image  of  R,  f(0R,  R)  ,  and  0r.  These  image  points  are  denoted  by  {(0r,  f(0R, 

R)).  In  practice,  f(0x,  X)  for  any  X  will  be  standardized  and  denoted  by  fx- 
The  second  coordinate,  f(0H,  R)  will  be  replaced  by  fg.  We  assume  that  these 
points  (the  images  of  fuzzy  response  patterns)  swarm  around  R,  and  that  {(0r, 
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fn) )  follow  a  bivariate  normal  distribution  (Tatsuoka  &  Tatsuoka,  1987; 
Tatsuoka,  1990),  called  a  "bug  distribution". 

The  cognitive  state  R  whose  6^  is  in  somewhere  between  -3  and  +3,  but 
for  which  the  absolute  value  of  fg  is  larger  than  3  may  not  really  exist 
(Tatsuoka,  1984).  If  the  values  of  f  for  some  states  are  close  to  zero,  many 
examinees  will  be  classified  into  such  states. 

The  mapping  by  f  may  not  be  one-to-one,  but  DiBello  and  Baillie  (1992) 
proved  that  f  is  indeed  almost  one-to-one  everywhere.  The  cases  for  the 
mapping  not  being  one-to-one  will  never  happen  when  the  IRT  parameters  aj  and 
bj  are  estimated  from  a  real  dataset.  The  standardized  f(x,0) ,  f,  will  be  the 
y-axis  of  the  classification  space,  called  Rule  Space  (Tatsuoka,  1985). 
However,  the  name  "Rule  Space"  may  be  misleading  because  the  mapped  cognitive 
states  can  be  misconception  states,  knowledge  states  or  even  be  personality 
states.  Tatsuoka  (1985)  showed  that  the  expectation  of  f(x,0)  is  zero  r  ,d  the 
variance  is  given  by  3 , 


Var[f(x.  0)]  Pj(0)Qj(0)(Pj(0)  -T(0))2 

i 


(3) 


The  configuration  in  rule  space  is  something  like  what  is  shown  in 
Figure  1. 

Insert  Figure  1  about  here 

In  this  figure,  the  ellipses  represent  equal  density  contours  for  the 
bug  distributions.  The  covariance  matrix  of  a  bug  distribution  will  be  a 
diagonal  matrix  with  the  variances  of  0  and  f  as  the  diagonal  elements  since 
these  variables  are  uncorrelated  (Tatsuoka,  1985). 
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4.  Classification  Procedure  Suppose  an  examinee ' s  response  patterns  are 
mapped  into  the  rule  space.  Then,  the  distance  between  the  individual 

examinee's  point,  (e,,,  Cx)  centroid  (Sr,  of  the  bug  distribution  R 

is  given  by  (4) ,  since  the  covariance  matrix  £  of  the  distribution  is  as  shown 
in  Equation  (5) . 

d2  -  (0^  -  eR)V(i/i(0R))  +  (fx  -  Cr)^.  (4) 

1/1(0^)  0 

0  1  (5) 

The  Mahalanobis  distance  (4)  follows  a  Chi-Square  distribution  with  two 
degrees  of  freedom  (Lachenbruch,  1975).  Suppose  an  examinee's  point  X  is 
classified  into  one  of  the  600  predetermined  groups  (or,  equivalently, 
knowledge  states)  determined  from  Table  1.3.3.  Then,  600  Mahalanobis  distances 
are  first  computed.  If  the  criterion  value  of  xz^  is  set  to  4.605  (p-.25), 
then  the  cognitive  states  whose  Mahalanobis  distance  from  X  is  less  than 
4.605  will  be  considered  as  eligible  cognitive  states  for  classification  of  X. 
If  there  is  no  cognitive  state  whose  Mahalanobis  distance  from  X  is  less  than 
4.605,  then  X  will  be  left  unclassified. 

Suppose  States  and  R2  are  tiie  two  closest  ones  to  X,  that  is,  which 
have  the  two  smallest  Mahalanobis  distances  from  X;  then  Bayes'  decision  rule 
for  minimum  error  will  be  applied  to  them  to  determine  the  final  group  for  X, 
and  the  total  classification  error  will  be  computed  (Tatsuoka  6e  Tatsuoka, 
1987).  If  the  covariance  matrices  of  two  states  are  almost  identical,  as  they 
are  in  cases  with  which  we  deal,  and  their  distributions  are  normal,  then  the 
Bayes'  decision  rule  becomes  equivalent  to  considering  a  linear  discriminant 
function.  That  is,  the  negative  of  the  logarithm  of  the  ratio  of  the 
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posterior  probabilities  of  and  Rj  for  X  will  be  a  linear  function  under  the 
normality  and  equal  covariances  conditions. 

Kim  (1990)  examined  the  effect  of  violation  of  the  normality  requirement 
with  simulated  data  in  the  rule  space,  and  found  that  the  linear  discriminant 
function  is  robust  against  this  violation.  Kim  further  compared  the 
classification  results  by  the  linear  discriminant  functions  and  K  nearest 
neighbors  method,  which  is  a  non-parametric  classification  approach  and  does 
not  assume  the  normality  of  a  bug  distribution,  and  found  that  the  linear 
discriminant  functions  performed  better. 

Suppose  R^  is  the  cognitive  state  to  which  X  belongs,  then  the  response 
pattern  X  and  the  ideal-item-score  pattern  for  R^  should  be  close  to  each 
other.  Since  R^  corresponds  to  an  attribute  mastery  pattern  Am,  the  response 
pattern  X  also  corresponds  to  Am  with  high  probability.  In  other  words,  the 
response  pattern  X  is  converted  to  the  attribute  mastery  pattern  corresponding 
to  Rj. 

Since  the  bug  distribution  for  Rj  is  assumed  to  be  bivariate  normal,  the 
posterior  probability  of  R^  given  X  can  be  computed  by  using  the  prior 
probability  of  R^,  as  discussed  in  Tatsuoka  &  Tatsuoka  (1987). 

5.  Multidimensional  Rule  Space  and  Generalized  Zetas  After  mapping  600 
ideal-item-score  patterns  into  the  Cartesian  Product  space  of  0  and  f,  the 
images  of  these  600  Ideal-item-score  patterns  may  become  too  close  and  too 
crowded,  that  is  they  may  be  too  densely  packed  on  the  plane  for 
classification  purposes.  If  the  mapped  cognitive  states  are  not  well 
E  arated,  then  the  error  rates  for  classification  become  unacceptably  large. 
In  order  to  separate  the  images  of  ideal-item-score  patterns ,  additional 


dinensions  may  be  needed.  For  the  analysis  of  SAT  M,  Section  2,  five 
dimensions  are  added. 

Generalized  fs  were  first  defined  by  Varadi  &  Tatsuoka  (1989).  Suppose  F 
is  a  subset  of  items,  then  the  generalized  fj.  is  defined  as  the  sum  of  the 
scalar  product  of  two  residuals,  (Pj(e)-Xj)'  {Pj(0)-T(6) ) ,  over  all  j  in  F, 
divided  by  the  standard  deviation  of  the  sum.  Selection  of  F  is  still  an  art 
and  its  further  development  is  left  as  a  research  topic  for  the  future. 
However,  it  is  recommended  to  take  union  and  intersection  sets  of  the  items 

which  correspond  to  the  attribute  row  vectors,  Aj . A^  of  the  incidence 

matrix.  Generalized  zeta  defined  on  the  items  involving  A^,  is  given  below 
with  its  numerator  function  f: 

f(z,e,)  -  (Pj(e,)-Zj,  Pj(e,)-T(e^))  (6) 

-  <Qi,'[Pj(e^)-Xj],  Qk'fPj(6x)-T(ex)]) 

-  Qk'(IPj(e,)-xj,  (p^(e,)-T(e,)]) 

r*  -  f(z,0,)/SQRT{Var[f(z.ej])  (7) 

where  z  -  Q^x,  and  0,  is  the  Maximum  Likelihood  Estimate  obtained  from  the 
items  involving 

The  expectation  and  variance  of  f(z,0,)  are  given  by  (8)  and  (9). 

Elf(z,0J]  «  0  (8) 

Var{f(z,  0,))  =  Pj(e^)  (1  -  Pj(0,))  [Pj(0,)  -  T(0,)]2  (9) 

j  «  {Q»>‘0} 

The  generalized  fs  are  uncorrelated  with  0,  which  can  be  shown  in 
exactly  in  the  same  manner  as  the  proof  for  the  uncorrelatedness  of  0  and  f 
given  in  Tatsuoka  (1985).  Furthermore,  a  generalized  f  computed  by  using  the 
items  involving  any  combination  of  A^  —  defined  as  the  union  or  intersection 
sets  of  Afc  ,k-l,..L  —  also  has  the  orthogonality  property  with  0. 
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Any  generalized  f  can  be  added  to  the  original  two-dimensional  Cartesian 
product  space  as  a  new  dimension,  and  a  multidimensional  classification  space 
can  be  formulated.  Both  the  ideal-item-score  patterns  and  examinees'  response 
patterns  are  mapped  into  the  (m+2)-  dimensional  Cartesian  product  space  {(6, 

f,  fj,  . Cm))-  The  larger  the  value  of  is,  the  more  unusual  the 

performance  on  the  items  involving  Attribute  is .  Thus ,  each  coordinate  in 
the  multidimensional  rule  space  can  maintain  interpretability . 

The  set  of  atoms  in  the  lattice  of  K  attributes  forms  a  basis  (Tatsuoka, 
1991,  Birkhoff,  1970),  but  it  is  very  difficult  to  give  intuitive 
interpretations  to  the  atoms  unless  the  incidence  matrix  is  diagonal  —  each 
attribute  being  involved  in  only  one  item  and  each  item  involving  only  one 
attribute.  So,  the  atoms  are  not  used  in  the  rule-space  model  although  they 
are  mathematically  useful  entities.  However,  if  intuitive  interpretations  of 
the  coordinates  are  not  required,  then  the  atoms  can  be  used  for  formulating  a 
multidimensional  space,  after  transforming  item  score  patterns. 

For  SAT  M,  Section  2,  five  generalized  fs  were  added  to  the  original 
two-dimensional  space,  and  classification  of  examinees  was  done  in  the 
resulting  seven  dimensional  space.  The  new  dimensions  are  shown  in  Table  A. 2. 

Insert  Table  A. 2  about  here 

The  interpretation  of  each  new  axis  is  similar  to  that  of  f  which  uses 
all  the  items.  For  example,  is  computed  using  the  items  involving  the 
attributes  1,3  and  4  in  which  14  items  are  considered.  If  the  value  of  is 
large,  then  the  pattern  of  the  14  relevant  items  is  aberrant,  while  a  smaller 
value  (including  a  negative  value)  of  indicates  that  the  pattern  conforms 
well  to  the  order  of  difficulty  for  the  14  items. 
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The  bug  distributions  for  cognitive  states  —  the  images  of  the  ideal- 
item-score  patterns  and  their  fuzzy  response  patterns  into  the  m+2  dimensional 
classification  space  —  are  assumed  to  be  multivariate  normal  distributions . 
Their  centroids  are  the  images  of  the  ideal-item-score  patterns.  A  squared 
Mahalanobis  distance  between  X  and  the  image  of  R  that  is  the  centroid  of  bug 
distribution  R,  or  a  cognitive  state  R  follows  a  distribution  with  m+2 
degrees  of  freedom  (Lachenbruch,  1975).  The  classification  procedure  and 
computation  of  error  probabilities,  prior  and  posterior  probabilities  are  a 
straightforward  extension  of  the  two  dimensional  case. 

After  classifying  examinees'  response  patterns  into  one  of  the 
predetermined  groups  or  cognitive  states,  their  item  response  patterns 
correspond  to  the  attribute  mastery  patterns  along  with  the  information  about 
D2,  error  probabilities,  probability  of  belonging  to  the  cognitive  state  to 
which  the  examinees  are  classified,  ML  estimates  of  9,  f  and  generalized  fs 
(Varadi  &  Tatsuoka,  1989).  We  propose  to  use  the  attribute  patterns  to 
estimate  Attribute  Characteristic  Curves,  which  is  comparable  to  the 
estimation  of  Item  Response  Curves.  However,  we  don't  use  parametric  functions 
for  PFAs.  Non-par ame trie  estimation  of  PFAs  will  be  illustrated  with  the 
attribute  mastery  patterns  of  SAT  M  Section  4.  In  the  next  section,  analysis 
results  will  be  described. 


Table  A.l  The  first  10  out  of  600  ideal- item- score 

patterns  derived  from  the  incidence  matrix  given  in  Table  1. 


Cognitive  Ideal -Item- Score 
States  Patterns,  25  items 

e 

c 

Attribute 

Patterns 

1 

1111111111111111111111111 

5.00 

.52 

11111111111111 

2 

1111111111111111011111000 

1.83 

-1.37 

11111111110111 

3 

1111111111111011111101111 

2.30 

1.88 

11111101111111 

4 

1111111111111011011101000 

1.12 

-.55 

11111101110111 

5 

1111111111110111111111111 

3.06 

1.43 

11011111111111 

6 

1111111111110111011111000 

1.41 

-.59 

11011111110111 

7 

1111111111110011111101111 

1.74 

2.59 

11011101111111 

8 

1111111111110011011101000 

.82 

.14 

11011101110111 

9 

1111111110111111101111101 

1.98 

.67 

11111111011111 

10 

1111111110111111001111000 

1.14 

-.69 

11111111010111 

301  6iiiii66i66i66iiii6666666  -.ei 

302  0111110010011010100000000  -.60 


- . 94  10111011000111 

-.05  10111001001111 


591  0000000000000001100000000 

592  0000000000000001010000000 

593  0000000000000001000100000 

594  0000000000000001000000000 

595  0000000000000000110000010 

596  0000000000000000110000000 

597  0000000000000000100000000 

598  0000000000000000010000010 

599  0000000000000000010000000 

600  0000000000000000000000000 


-2.76 

1.52 

10011000011101 

-2.84 

1.42 

10011100110001 

-2.87 

1.38 

10001000010111 

-3.52 

.86 

10001000010001 

-2.44 

2.44 

00010101011101 

-2.74 

1.76 

00010100111101 

-3.32 

1.13 

00010000001101 

-2.91 

1.65 

00010101111001 

-3.48 

1.02 

00010100110001 

-5.00 

.53 

00000000000000 

61 


Table  A. 2  The  Generalized  Cs  Added  as  New  Dimensions  and 
Their  Attribute  Sub  Space 


Attributes 

Corresponding  Items 

1 

C 1  Aj+As+A* 

2.3,4.5,8.9,10.11.13.15.16,19.20.25 

2 

Ca  Aj 

2.6.13,21 

3 

C3  Ag+Ajo 

8,11,12.16,18,20.22.23,25 

4 

C*  Aii+Ai2 

1.2,5,6.8.9,11.12.13.15.16,19,20,21,22,23 

5 

Zi  An 

14,17 
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H«r«  is  the  progression  of  the  student's  points  throughout 
the  test.  “o"  ■  finsl  point. 


X  ■  ability  level,  Y  >  unusual ness  of  response  pattern 
Press  HELP  for  more  information 


Figure  1  An  Exanple  of  the  Rule  Space  configuration 
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Append i x  I  I 

Possible  Score  Reports  Based  on  the  Rule-Space  Results 

Potential  Audiences/Usages  and  Types  of  Reports 

Audience 

Usage 

Type  cf  Report* 

Higher  educadm  institutions 

selection,  placement  of  applicants 

per  examinee:  13 

High  schools 

a.  Testodcers 

vocational  decisions;  skills  to  be  imiKOved 

per  examinee: 

\X  3/4,  5 

b.  Teachers 

remediation^ture  instruction  planning 

per  class:  8, 9 

c.  Principals 

teacher/curriculum  evaluation 

entires  schocd^ 
class  conmarisons: 

6,7 

d.  District  Administration 

school/curriculum  evaluation,  educational 

entire  (fistrict  *  adKx>l 

pctiicy 

conq>arisons:  6, 7 

e.  State  Administration 

district/curriculum  evaluation,  educational 

entire  state  district 

policy 

comparisons:  6, 7 

Item  developers 

test  evaluation:  items  to  be  improved/ 

per  item:  10, 11, 12 

arided/ddeted 

13, 14 

•  For  key  see  attached  list  of  report's  components 

Report's  Components 

1 .  SAT  scale  score 

2.  SAT  percentile  score  (relative  standing  school-wise/nadon-wise) 

3.  Attribute  probability  profile  per  student 

4.  Attribute  probabOity  profile  on  a  6  point  scale,  where:  A«90-100;  B«70-89;  OSO- 
69;  D*3(M9;  E-10-29;  F=0-9. 

5.  Detailed  diagnosis  (nanative)  [descri|}tion  of  attribute-  mastery  profile; 
appropriateness  scores;  recommendations . . . ) 


6.  Attribute  profile -means 

7.  SAT  scores  -  iheans 

8.  S-A  (Student-Attribute)  chan 

9.  S-I  (Student -Item)  chan 


10.  IRT  item  difficulty  index 

11.  IRT  item  discrimination  intfex 

12.  Attribute  pattern  per  item  (Q  matrix) 

13.  Reliability  in^ces 

14.  Results  of  regressing  item  difficulties  cm  attribute  veactors. 
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The  Database  for  the  Retrieval  System 


a.  Psvc 


c  data 


The  basic  information  available  for  enhancing  scoring  reports  is  stored  in  the  database.  The 
database  consists  of  fcnir  parts: 


1 .  The  score  matrix  which  contains  student  ID,  an  item  req;)onse  pattern,  a  6-value,  a  C- 

value  (index  for  unusualness  of  a  pattern),  an  attribute  pattern  for  examinees. 

2.  The  incidence  matrix. 

3.  The  probability  matrix  of  indicating  each  item’s  success  rate  at  var''us  6-levels  (will  be 
converted  to  SAT  scale  later), 

4.  The  probability  matrix  of  indicating  each  attribute's  mastery  rate  at  various  6  levels. 


b.  C 


laUlaia 


1.  Demographic  data:  student's  gender,  ethnicity,  SES  . . . 

2.  Studei^sclasstoom/school/di^cVstateaffiliaticm 

3.  Test  format ... 


The  infonnation  stored  in  the  database  will  be  available  foe  creating  a  variety  of 
combinations,  according  to  a  request  by  a  user. 

A  mapping  sentence  that  containing  content,  process  and  context  facets  areas  will  be 
available  to  help  choosing  any  combinadon  of  variables.  Some  users  may  chose  content 
variables  for  making  a  summary  stadsdes  of  item  and  attribute  performance  on  a  test  while 
others  may  select  context  (forms  and  setdngs  of  tests)  variables  to  see  their  effea  on 
performance  differences. 

The  retrieval  system  extracts  any  combinadon  of  informadon  on  the  variables  from  the 
database  and  prepares  a  summary  for  the  required  report. 
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